[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-7.9-debug (green@centos7-base) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Sat Mar 26 23:28:42 EDT 2022 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000013edfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] AGP: No AGP bridge found [ 0.000000] e820: last_pfn = 0x13ee00 max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: write-back [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF uncachable [ 0.000000] C0000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 00C0000000 mask FFC0000000 uncachable [ 0.000000] 1 disabled [ 0.000000] 2 disabled [ 0.000000] 3 disabled [ 0.000000] 4 disabled [ 0.000000] 5 disabled [ 0.000000] 6 disabled [ 0.000000] 7 disabled [ 0.000000] PAT configuration [0-7]: WB WC UC- UC WB WP UC- UC [ 0.000000] e820: last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] mapped at [ffffffffff200b30] [ 0.000000] Base memory trampoline at [ffff880000099000] 99000 size 24576 [ 0.000000] BRK [0x023c1000, 0x023c1fff] PGTABLE [ 0.000000] BRK [0x023c2000, 0x023c2fff] PGTABLE [ 0.000000] BRK [0x023c3000, 0x023c3fff] PGTABLE [ 0.000000] BRK [0x023c4000, 0x023c4fff] PGTABLE [ 0.000000] BRK [0x023c5000, 0x023c5fff] PGTABLE [ 0.000000] BRK [0x023c6000, 0x023c6fff] PGTABLE [ 0.000000] RAMDISK: [mem 0xbc2e2000-0xbffbffff] [ 0.000000] Early table checksum verification disabled [ 0.000000] ACPI: RSDP 00000000000f5950 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 00000000bffe1bb7 00034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 00000000bffe1a53 00074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 00000000bffe0040 01A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 00000000bffe0000 00040 [ 0.000000] ACPI: APIC 00000000bffe1ac7 00090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 00000000bffe1b57 00038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 00000000bffe1b8f 00028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000013edfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x13e5e3000-0x13e609fff] [ 0.000000] Reserving 128MB of memory at 768MB for crashkernel (System RAM: 4077MB) [ 0.000000] kvm-clock: cpu 0, msr 1:3e592001, primary cpu clock [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 358900760 cycles [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal [mem 0x100000000-0x13edfffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009efff] [ 0.000000] node 0: [mem 0x00100000-0xbffcdfff] [ 0.000000] node 0: [mem 0x100000000-0x13edfffff] [ 0.000000] Initmem setup node 0 [mem 0x00001000-0x13edfffff] [ 0.000000] On node 0 totalpages: 1043820 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 21 pages reserved [ 0.000000] DMA zone: 3998 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 12224 pages used for memmap [ 0.000000] DMA32 zone: 782286 pages, LIFO batch:31 [ 0.000000] Normal zone: 4024 pages used for memmap [ 0.000000] Normal zone: 257536 pages, LIFO batch:31 [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ5 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] ACPI: IRQ10 used by override. [ 0.000000] ACPI: IRQ11 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] e820: [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 38 pages/cpu s115176 r8192 d32280 u524288 [ 0.000000] pcpu-alloc: s115176 r8192 d32280 u524288 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 0 1 2 3 [ 0.000000] KVM setup async PF for cpu 0 [ 0.000000] kvm-stealtime: cpu 0, msr 13e2135c0 [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 1027487 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] audit: disabled (until reboot) [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] AGP: Checking aperture... [ 0.000000] AGP: No AGP bridge found [ 0.000000] Memory: 3820268k/5224448k available (8172k kernel code, 1049168k absent, 355012k reserved, 5773k data, 2532k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=4. [ 0.000000] Offload RCU callbacks from all CPUs [ 0.000000] Offload RCU callbacks from CPUs: 0-3. [ 0.000000] NR_IRQS:327936 nr_irqs:456 0 [ 0.000000] Console: colour *CGA 80x25 [ 0.000000] console [ttyS1] enabled [ 0.000000] allocated 25165824 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ODEBUG: 73 of 73 active objects replaced [ 0.000000] hpet clockevent registered [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.435636] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.437271] pid_max: default: 32768 minimum: 301 [ 0.438338] Security Framework initialized [ 0.438947] SELinux: Initializing. [ 0.439496] SELinux: Starting in permissive mode [ 0.440667] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes) [ 0.443208] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.444998] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.446368] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.448265] Initializing cgroup subsys memory [ 0.449267] Initializing cgroup subsys devices [ 0.450264] Initializing cgroup subsys freezer [ 0.451202] Initializing cgroup subsys net_cls [ 0.452073] Initializing cgroup subsys blkio [ 0.453052] Initializing cgroup subsys perf_event [ 0.453987] Initializing cgroup subsys hugetlb [ 0.454636] Initializing cgroup subsys pids [ 0.455573] Initializing cgroup subsys net_prio [ 0.456788] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.457939] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.459036] tlb_flushall_shift: 6 [ 0.459655] FEATURE SPEC_CTRL Not Present [ 0.460395] FEATURE IBPB_SUPPORT Not Present [ 0.461384] Spectre V2 : Vulnerable [ 0.462102] Speculative Store Bypass: Vulnerable [ 0.463350] debug: unmapping init [mem 0xffffffff82019000-0xffffffff8201ffff] [ 0.469431] ACPI: Core revision 20130517 [ 0.471585] ACPI: All ACPI Tables successfully acquired [ 0.472895] ftrace: allocating 30294 entries in 119 pages [ 0.518252] Enabling x2apic [ 0.519238] Enabled x2apic [ 0.520478] Switched APIC routing to physical x2apic. [ 0.523015] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.524941] smpboot: CPU0: Intel QEMU Virtual CPU version 2.5+ (fam: 0f, model: 6b, stepping: 01) [ 0.628893] Performance Events: unsupported Netburst CPU model 107 no PMU driver, software events only. [ 0.646228] kvm-clock: cpu 1, msr 1:3e592041, secondary cpu clock [ 0.650715] KVM setup async PF for cpu 1 [ 0.651966] kvm-stealtime: cpu 1, msr 13e2935c0 [ 0.665419] kvm-clock: cpu 2, msr 1:3e592081, secondary cpu clock [ 0.670430] KVM setup async PF for cpu 2 [ 0.633940] smpboot: Booting Node 0, Processors #1 #2 #3 OK [ 0.672049] kvm-stealtime: cpu 2, msr 13e3135c0 [ 0.681812] kvm-clock: cpu 3, msr 1:3e5920c1, secondary cpu clock [ 0.686113] Brought up 4 CPUs [ 0.686138] KVM setup async PF for cpu 3 [ 0.686146] kvm-stealtime: cpu 3, msr 13e3935c0 [ 0.689803] smpboot: Max logical packages: 1 [ 0.691122] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.696758] devtmpfs: initialized [ 0.697984] x86/mm: Memory block size: 128MB [ 0.702852] EVM: security.selinux [ 0.703780] EVM: security.ima [ 0.704580] EVM: security.capability [ 0.708138] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.710413] NET: Registered protocol family 16 [ 0.719249] cpuidle: using governor haltpoll [ 0.722206] ACPI: bus type PCI registered [ 0.723562] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.725535] PCI: Using configuration type 1 for base access [ 0.734254] ACPI: Added _OSI(Module Device) [ 0.735537] ACPI: Added _OSI(Processor Device) [ 0.736895] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.738388] ACPI: Added _OSI(Processor Aggregator Device) [ 0.739981] ACPI: Added _OSI(Linux-Dell-Video) [ 0.742064] ACPI: EC: Look up EC in DSDT [ 0.744694] ACPI: Interpreter enabled [ 0.745469] ACPI: (supports S0 S3 S4 S5) [ 0.746465] ACPI: Using IOAPIC for interrupt routing [ 0.748173] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.750591] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.757420] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.759331] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.760824] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.762080] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.764638] acpiphp: Slot [2] registered [ 0.765839] acpiphp: Slot [3] registered [ 0.767049] acpiphp: Slot [4] registered [ 0.768352] acpiphp: Slot [5] registered [ 0.769808] acpiphp: Slot [6] registered [ 0.771089] acpiphp: Slot [7] registered [ 0.772166] acpiphp: Slot [8] registered [ 0.773548] acpiphp: Slot [9] registered [ 0.774891] acpiphp: Slot [10] registered [ 0.776248] acpiphp: Slot [11] registered [ 0.777510] acpiphp: Slot [12] registered [ 0.778884] acpiphp: Slot [13] registered [ 0.780213] acpiphp: Slot [14] registered [ 0.781534] acpiphp: Slot [15] registered [ 0.782915] acpiphp: Slot [16] registered [ 0.784215] acpiphp: Slot [17] registered [ 0.785534] acpiphp: Slot [18] registered [ 0.786717] acpiphp: Slot [19] registered [ 0.787854] acpiphp: Slot [20] registered [ 0.789032] acpiphp: Slot [21] registered [ 0.790270] acpiphp: Slot [22] registered [ 0.791482] acpiphp: Slot [23] registered [ 0.792714] acpiphp: Slot [24] registered [ 0.794056] acpiphp: Slot [25] registered [ 0.795954] acpiphp: Slot [26] registered [ 0.797202] acpiphp: Slot [27] registered [ 0.798673] acpiphp: Slot [28] registered [ 0.800120] acpiphp: Slot [29] registered [ 0.801592] acpiphp: Slot [30] registered [ 0.803039] acpiphp: Slot [31] registered [ 0.804320] PCI host bridge to bus 0000:00 [ 0.805647] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.807903] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.810148] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.812430] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.814636] pci_bus 0000:00: root bus resource [mem 0x140000000-0x1bfffffff window] [ 0.816354] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.817326] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.817868] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.818645] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.823350] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.825596] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.829233] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.831905] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.833882] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.836105] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.836732] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.839228] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.842078] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.844109] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.853644] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.855836] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.856892] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.861234] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.865380] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.886881] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.892464] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.901381] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.907026] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.920964] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.929900] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.938997] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.949230] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.959316] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.973519] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.986052] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.994369] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 1.021724] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 1.028210] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 1.034137] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 1.039199] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 1.061487] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 1.071287] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 1.081492] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 1.089242] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.109738] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 1.121037] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 1.122709] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 1.126494] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 1.129880] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 1.132414] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 1.136405] vgaarb: loaded [ 1.137470] SCSI subsystem initialized [ 1.138623] ACPI: bus type USB registered [ 1.139990] usbcore: registered new interface driver usbfs [ 1.141795] usbcore: registered new interface driver hub [ 1.143813] usbcore: registered new device driver usb [ 1.147958] PCI: Using ACPI for IRQ routing [ 1.149259] PCI: pci_cache_line_size set to 64 bytes [ 1.149540] e820: reserve RAM buffer [mem 0x0009fc00-0x0009ffff] [ 1.149546] e820: reserve RAM buffer [mem 0xbffce000-0xbfffffff] [ 1.149548] e820: reserve RAM buffer [mem 0x13ee00000-0x13fffffff] [ 1.149879] NetLabel: Initializing [ 1.150880] NetLabel: domain hash size = 128 [ 1.152161] NetLabel: protocols = UNLABELED CIPSOv4 [ 1.153544] NetLabel: unlabeled traffic allowed by default [ 1.155405] HPET: 3 timers in total, 0 timers will be used for per-cpu timer [ 1.157700] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.159330] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.165415] amd_nb: Cannot enumerate AMD northbridges [ 1.167650] Switched to clocksource kvm-clock [ 1.196763] pnp: PnP ACPI init [ 1.198026] ACPI: bus type PNP registered [ 1.201254] pnp 00:00: Plug and Play ACPI device, IDs PNP0303 (active) [ 1.201373] pnp 00:01: Plug and Play ACPI device, IDs PNP0f13 (active) [ 1.201435] pnp 00:02: [dma 2] [ 1.201487] pnp 00:02: Plug and Play ACPI device, IDs PNP0700 (active) [ 1.201632] pnp 00:03: Plug and Play ACPI device, IDs PNP0501 (active) [ 1.202245] pnp 00:04: Plug and Play ACPI device, IDs PNP0501 (active) [ 1.202341] pnp 00:05: Plug and Play ACPI device, IDs PNP0b00 (active) [ 1.203117] pnp: PnP ACPI: found 6 devices [ 1.205257] ACPI: bus type PNP unregistered [ 1.221694] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.221702] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.221705] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.221708] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.221710] pci_bus 0000:00: resource 8 [mem 0x140000000-0x1bfffffff window] [ 1.221799] NET: Registered protocol family 2 [ 1.223910] TCP established hash table entries: 32768 (order: 6, 262144 bytes) [ 1.226767] TCP bind hash table entries: 32768 (order: 8, 1048576 bytes) [ 1.230036] TCP: Hash tables configured (established 32768 bind 32768) [ 1.232041] TCP: reno registered [ 1.234490] UDP hash table entries: 2048 (order: 5, 196608 bytes) [ 1.237460] UDP-Lite hash table entries: 2048 (order: 5, 196608 bytes) [ 1.240221] NET: Registered protocol family 1 [ 1.242194] RPC: Registered named UNIX socket transport module. [ 1.243947] RPC: Registered udp transport module. [ 1.245127] RPC: Registered tcp transport module. [ 1.246327] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.247966] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.249458] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.251147] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.253244] PCI: CLS 0 bytes, default 64 [ 1.253431] Unpacking initramfs... [ 2.649891] debug: unmapping init [mem 0xffff8800bc2e2000-0xffff8800bffbffff] [ 2.653271] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.654915] software IO TLB [mem 0xb82e2000-0xbc2e2000] (64MB) mapped at [ffff8800b82e2000-ffff8800bc2e1fff] [ 2.659565] cryptomgr_test (52) used greatest stack depth: 14480 bytes left [ 2.660108] futex hash table entries: 1024 (order: 4, 65536 bytes) [ 2.660155] Initialise system trusted keyring [ 2.693029] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 2.700658] zpool: loaded [ 2.701729] zbud: loaded [ 2.703829] VFS: Disk quotas dquot_6.6.0 [ 2.705730] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 2.708691] NFS: Registering the id_resolver key type [ 2.710802] Key type id_resolver registered [ 2.712163] Key type id_legacy registered [ 2.713294] nfs4filelayout_init: NFSv4 File Layout Driver Registering... [ 2.716808] Key type big_key registered [ 2.717980] SELinux: Registering netfilter hooks [ 2.723501] cryptomgr_test (58) used greatest stack depth: 14048 bytes left [ 2.725654] cryptomgr_test (59) used greatest stack depth: 13968 bytes left [ 2.728012] NET: Registered protocol family 38 [ 2.729546] Key type asymmetric registered [ 2.730625] Asymmetric key parser 'x509' registered [ 2.732084] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250) [ 2.734331] io scheduler noop registered [ 2.735510] io scheduler deadline registered (default) [ 2.737215] io scheduler cfq registered [ 2.738331] io scheduler mq-deadline registered [ 2.739585] io scheduler kyber registered [ 2.744469] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 2.745766] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 2.748333] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.750436] ACPI: Power Button [PWRF] [ 2.751920] GHES: HEST is not enabled! [ 2.800714] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 10 [ 2.850678] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 11 [ 2.945429] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 [ 2.993498] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 10 [ 3.095746] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.124912] 00:03: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A [ 3.152621] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 3.157200] Non-volatile memory driver v1.3 [ 3.159166] Linux agpgart interface v0.103 [ 3.161176] crash memory driver: version 1.1 [ 3.163417] nbd: registered device at major 43 [ 3.168554] virtio-pci 0000:00:05.0: irq 24 for MSI/MSI-X [ 3.168603] virtio-pci 0000:00:05.0: irq 25 for MSI/MSI-X [ 3.168642] virtio-pci 0000:00:05.0: irq 26 for MSI/MSI-X [ 3.168687] virtio-pci 0000:00:05.0: irq 27 for MSI/MSI-X [ 3.168725] virtio-pci 0000:00:05.0: irq 28 for MSI/MSI-X [ 3.175441] virtio_blk virtio1: [vda] 66728 512-byte logical blocks (34.1 MB/32.5 MiB) [ 3.182640] virtio-pci 0000:00:06.0: irq 29 for MSI/MSI-X [ 3.182678] virtio-pci 0000:00:06.0: irq 30 for MSI/MSI-X [ 3.182719] virtio-pci 0000:00:06.0: irq 31 for MSI/MSI-X [ 3.182751] virtio-pci 0000:00:06.0: irq 32 for MSI/MSI-X [ 3.182783] virtio-pci 0000:00:06.0: irq 33 for MSI/MSI-X [ 3.189825] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.197082] virtio-pci 0000:00:07.0: irq 34 for MSI/MSI-X [ 3.197134] virtio-pci 0000:00:07.0: irq 35 for MSI/MSI-X [ 3.197177] virtio-pci 0000:00:07.0: irq 36 for MSI/MSI-X [ 3.197216] virtio-pci 0000:00:07.0: irq 37 for MSI/MSI-X [ 3.197246] virtio-pci 0000:00:07.0: irq 38 for MSI/MSI-X [ 3.203993] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.214953] virtio-pci 0000:00:08.0: irq 39 for MSI/MSI-X [ 3.214994] virtio-pci 0000:00:08.0: irq 40 for MSI/MSI-X [ 3.215017] virtio-pci 0000:00:08.0: irq 41 for MSI/MSI-X [ 3.215039] virtio-pci 0000:00:08.0: irq 42 for MSI/MSI-X [ 3.215120] virtio-pci 0000:00:08.0: irq 43 for MSI/MSI-X [ 3.224071] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.234016] virtio-pci 0000:00:09.0: irq 44 for MSI/MSI-X [ 3.234100] virtio-pci 0000:00:09.0: irq 45 for MSI/MSI-X [ 3.234136] virtio-pci 0000:00:09.0: irq 46 for MSI/MSI-X [ 3.234166] virtio-pci 0000:00:09.0: irq 47 for MSI/MSI-X [ 3.234201] virtio-pci 0000:00:09.0: irq 48 for MSI/MSI-X [ 3.240092] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.245196] virtio-pci 0000:00:0a.0: irq 49 for MSI/MSI-X [ 3.245240] virtio-pci 0000:00:0a.0: irq 50 for MSI/MSI-X [ 3.245279] virtio-pci 0000:00:0a.0: irq 51 for MSI/MSI-X [ 3.245317] virtio-pci 0000:00:0a.0: irq 52 for MSI/MSI-X [ 3.245351] virtio-pci 0000:00:0a.0: irq 53 for MSI/MSI-X [ 3.251425] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.257848] rdac: device handler registered [ 3.259314] hp_sw: device handler registered [ 3.260536] emc: device handler registered [ 3.262131] libphy: Fixed MDIO Bus: probed [ 3.267210] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 3.268985] ehci-pci: EHCI PCI platform driver [ 3.270511] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 3.272228] ohci-pci: OHCI PCI platform driver [ 3.273501] uhci_hcd: USB Universal Host Controller Interface driver [ 3.275284] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.278506] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.280462] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.284860] mousedev: PS/2 mouse device common for all mice [ 3.288618] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.290756] rtc_cmos 00:05: RTC can wake from S4 [ 3.292010] rtc_cmos 00:05: rtc core: registered rtc_cmos as rtc0 [ 3.293178] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.293694] kvm: host does not support poll control [ 3.293695] kvm: host upgrade recommended [ 3.297234] hidraw: raw HID events driver (C) Jiri Kosina [ 3.297484] usbcore: registered new interface driver usbhid [ 3.297485] usbhid: USB HID core driver [ 3.297574] drop_monitor: Initializing network drop monitor service [ 3.297632] Netfilter messages via NETLINK v0.30. [ 3.297708] TCP: cubic registered [ 3.297715] Initializing XFRM netlink socket [ 3.297968] NET: Registered protocol family 10 [ 3.298675] NET: Registered protocol family 17 [ 3.298719] Key type dns_resolver registered [ 3.299204] mce: Using 10 MCE banks [ 3.302327] Loading compiled-in X.509 certificates [ 3.304182] Loaded X.509 cert 'Magrathea: Glacier signing key: e34d0e1b7fcf5b414cce75d36d8482945c781ed6' [ 3.304278] registered taskstats version 1 [ 3.306869] modprobe (71) used greatest stack depth: 13456 bytes left [ 3.310444] Key type trusted registered [ 3.316973] Key type encrypted registered [ 3.317025] IMA: No TPM chip found, activating TPM-bypass! (rc=-19) [ 3.320410] BERT: Boot Error Record Table support is disabled. Enable it by using bert_enable as kernel parameter. [ 3.322244] rtc_cmos 00:05: setting system clock to 2024-02-27 17:08:18 UTC (1709053698) [ 3.336983] debug: unmapping init [mem 0xffffffff81da0000-0xffffffff82018fff] [ 3.339821] Write protecting the kernel read-only data: 12288k [ 3.342050] debug: unmapping init [mem 0xffff8800017fe000-0xffff8800017fffff] [ 3.344257] debug: unmapping init [mem 0xffff880001b9b000-0xffff880001bfffff] [ 3.355070] random: systemd: uninitialized urandom read (16 bytes read) [ 3.358824] random: systemd: uninitialized urandom read (16 bytes read) [ 3.361104] random: systemd: uninitialized urandom read (16 bytes read) [ 3.364956] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 3.370384] systemd[1]: Detected virtualization kvm. [ 3.372106] systemd[1]: Detected architecture x86-64. [ 3.373304] systemd[1]: Running in initial RAM disk. [ 3.378760] systemd[1]: No hostname configured. [ 3.380104] systemd[1]: Set hostname to . [ 3.382163] random: systemd: uninitialized urandom read (16 bytes read) [ 3.384412] systemd[1]: Initializing machine ID from random generator. [ 3.436893] dracut-rootfs-g (86) used greatest stack depth: 13264 bytes left [ 3.440129] random: systemd: uninitialized urandom read (16 bytes read) [ 3.442417] random: systemd: uninitialized urandom read (16 bytes read) [ 3.444300] random: systemd: uninitialized urandom read (16 bytes read) [ 3.446623] random: systemd: uninitialized urandom read (16 bytes read) [ 3.450090] random: systemd: uninitialized urandom read (16 bytes read) [ 3.452434] random: systemd: uninitialized urandom read (16 bytes read) [ 3.461922] systemd[1]: Reached target Swap. [ 3.465607] systemd[1]: Created slice Root Slice. [ 3.469637] systemd[1]: Listening on udev Control Socket. [ 3.474032] systemd[1]: Reached target Timers. [ 3.478017] systemd[1]: Reached target Local File Systems. [ 3.481981] systemd[1]: Listening on udev Kernel Socket. [ 3.485688] systemd[1]: Listening on Journal Socket. [ 3.488884] systemd[1]: Reached target Sockets. [ 3.492287] systemd[1]: Created slice System Slice. [ 3.496555] systemd[1]: Starting Journal Service... [ 3.499897] systemd[1]: Reached target Slices. [ 3.505877] systemd[1]: Starting dracut cmdline hook... [ 3.512467] systemd[1]: Starting Create list of required static device nodes for the current kernel... [ 3.518565] systemd-cgroups (106) used greatest stack depth: 13168 bytes left [ 3.525432] systemd[1]: Starting Load Kernel Modules... [ 3.530383] systemd[1]: Starting Setup Virtual Console... [ 3.535306] systemd[1]: Started Journal Service. [ 3.554367] random: fast init done [ 3.659635] tsc: Refined TSC clocksource calibration: 2399.954 MHz [ 3.989398] libata version 3.00 loaded. [ 3.992062] ata_piix 0000:00:01.1: version 2.13 [ 4.004824] scsi host0: ata_piix [ 4.009725] scsi host1: ata_piix [ 4.011530] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.013240] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 4.070948] ip (319) used greatest stack depth: 13080 bytes left [ 4.105386] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [ 4.106770] ip (342) used greatest stack depth: 12464 bytes left [ 4.165439] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input2 [ 4.267636] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready [ 8.512165] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ 9.103583] systemd-journald[100]: Received SIGTERM from PID 1 (systemd). [ 9.465851] SELinux: Disabled at runtime. [ 9.468132] SELinux: Unregistering netfilter hooks [ 9.569817] ip_tables: (C) 2000-2006 Netfilter Core Team [ 9.574552] systemd[1]: Inserted module 'ip_tables' [ 10.349691] systemd-journald[566]: Received request to flush runtime journal from PID 1 [ 10.564063] input: PC Speaker as /devices/platform/pcspkr/input/input3 [ 10.641320] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 10.903107] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 10.906237] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ 11.603352] cp (688) used greatest stack depth: 11824 bytes left [ 13.970022] mount.nfs (736) used greatest stack depth: 10728 bytes left [ 21.788509] device-mapper: uevent: version 1.0.3 [ 21.791025] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 26.170916] libcfs: loading out-of-tree module taints kernel. [ 26.172855] libcfs: module verification failed: signature and/or required key missing - tainting kernel [ 31.045363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 31.060616] alg: No test for adler32 (adler32-zlib) [ 31.317619] PCLMULQDQ-NI instructions are not detected. [ 31.933952] Lustre: Lustre: Build Version: 2.15.61_76_g2ee32a4 [ 32.095549] LNet: Added LNI 192.168.204.153@tcp [8/256/0/180] [ 32.097253] LNet: Accept secure, port 988 [ 33.638737] Key type lgssc registered [ 33.952137] Lustre: Echo OBD driver; http://www.lustre.org/ [ 36.877760] icp: module license 'CDDL' taints kernel. [ 36.879081] Disabling lock debugging due to kernel taint [ 39.282103] ZFS: Loaded module v0.8.6-1, ZFS pool version 5000, ZFS filesystem version 5 [ 42.199103] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 46.725547] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 48.535056] LDISKFS-fs (vde): file extents enabled, maximum tree depth=5 [ 48.540967] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 50.459215] LDISKFS-fs (vdf): file extents enabled, maximum tree depth=5 [ 50.463719] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 53.148815] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing load_modules_local [ 55.892639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 55.915786] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 55.921684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 56.988846] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 56.999501] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 57.030438] Lustre: lustre-MDT0000: new disk, initializing [ 57.049279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 57.056574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 57.086658] mount.lustre (6494) used greatest stack depth: 10136 bytes left [ 57.796206] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 61.953724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 61.974621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 61.997987] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 62.006942] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 62.009145] Lustre: Skipped 1 previous similar message [ 62.040115] Lustre: lustre-MDT0001: new disk, initializing [ 62.053599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 62.060795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 62.064012] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 62.814953] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 66.824679] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 66.828563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 66.845839] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 66.848276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 66.896880] Lustre: lustre-OST0000: new disk, initializing [ 66.898261] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 66.908346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 68.039415] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 70.498483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 70.501373] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 70.513703] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 72.085815] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 72.089019] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 72.104718] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 72.108766] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 72.134635] Lustre: lustre-OST0001: new disk, initializing [ 72.136691] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 72.148612] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 73.275997] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 77.538738] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 79.458408] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 79.460493] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 79.468941] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 84.692311] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 90.367869] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing check_logdir /tmp/testlogs/ [ 90.930978] random: crng init done [ 91.230494] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing yml_node [ 92.239702] Lustre: DEBUG MARKER: Client: 2.15.61.76 [ 92.839829] Lustre: DEBUG MARKER: MDS: 2.15.61.76 [ 93.446228] Lustre: DEBUG MARKER: OSS: 2.15.61.76 [ 93.818160] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Feb 27 12:09:49 EST 2024 [ 95.203795] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball 110 [ 95.543571] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 107.123988] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 107.127859] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 107.132995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 109.507957] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 109.507984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 109.516293] Lustre: Skipped 1 previous similar message [ 112.325664] Lustre: server umount lustre-MDT0000 complete [ 113.354169] LustreError: 6504:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709053808 with bad export cookie 13699269355392084106 [ 113.355285] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 113.364200] LustreError: 6504:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 114.515931] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 114.515988] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 114.515990] Lustre: Skipped 1 previous similar message [ 114.516092] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 114.516094] Lustre: Skipped 3 previous similar messages [ 114.531089] LustreError: Skipped 1 previous similar message [ 118.523956] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 118.523957] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 118.523962] Lustre: Skipped 1 previous similar message [ 118.532062] LustreError: Skipped 1 previous similar message [ 119.451831] Lustre: server umount lustre-MDT0001 complete [ 126.424286] Lustre: server umount lustre-OST0000 complete [ 133.426480] Lustre: server umount lustre-OST0001 complete [ 134.665130] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_hostid [ 136.022207] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing load_modules_local [ 137.968739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 139.417093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 140.785231] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 140.789307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 142.219231] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 142.222580] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 143.915426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 143.935046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 143.992476] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 144.000206] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 144.030781] Lustre: lustre-MDT0000: new disk, initializing [ 144.049812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 144.055224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 144.077580] mount.lustre (15873) used greatest stack depth: 10120 bytes left [ 144.794119] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 148.042790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 148.061556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 148.086678] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 148.155111] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 148.177511] mount.lustre (16820) used greatest stack depth: 9560 bytes left [ 148.871005] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 151.124968] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 151.699860] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 153.255073] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 153.258538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 153.277058] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 153.280329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 153.339974] Lustre: lustre-OST0000: new disk, initializing [ 153.341087] Lustre: Skipped 1 previous similar message [ 153.342237] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 153.343736] Lustre: Skipped 2 previous similar messages [ 153.357964] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 153.360431] Lustre: Skipped 1 previous similar message [ 154.503235] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 154.890988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 154.894169] Lustre: Skipped 1 previous similar message [ 154.896041] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 154.911902] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 156.706959] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 157.414791] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 157.461696] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 158.189226] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 158.230708] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 159.908267] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 159.912131] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 159.916399] Lustre: Skipped 1 previous similar message [ 159.918504] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 159.920579] Lustre: Skipped 1 previous similar message [ 164.705065] Lustre: server umount lustre-OST0000 complete [ 168.164188] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 168.166330] LustreError: Skipped 1 previous similar message [ 168.167585] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 168.172055] Lustre: Skipped 1 previous similar message [ 168.174294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 168.176217] Lustre: Skipped 1 previous similar message [ 171.647586] Lustre: server umount lustre-MDT0000 complete [ 172.549295] LustreError: 15883:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709053867 with bad export cookie 13699269355392085450 [ 172.550229] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 172.556745] LustreError: 15883:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 172.635479] Lustre: server umount lustre-MDT0001 complete [ 174.629649] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 12:11:10 (1709053870) [ 176.193209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 176.291412] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 176.304780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 177.025208] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 179.074189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 179.868134] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 180.817298] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 181.335124] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 182.854735] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 182.857858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 184.027342] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 184.926739] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 191.439224] Lustre: Modifying parameter general.osc.*.max_dirty_mb in log params [ 194.164132] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 194.166109] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 194.169283] Lustre: Skipped 1 previous similar message [ 194.170829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 194.172462] Lustre: Skipped 2 previous similar messages [ 196.162821] Lustre: server umount lustre-MDT0000 complete [ 197.037326] LustreError: 20658:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709053892 with bad export cookie 13699269355392086304 [ 197.039031] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 197.044495] LustreError: 20658:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 197.126340] Lustre: server umount lustre-MDT0001 complete [ 208.057726] Lustre: 23835:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1709053893/real 1709053893] req@ffff8800a01daa00 x1792072720900160/t0(0) o39->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1709053903 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 218.072084] Lustre: 23835:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1709053903/real 1709053903] req@ffff8800a01daa00 x1792072720900480/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1709053913 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 218.223575] Lustre: server umount lustre-OST0000 complete [ 225.998192] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing load_modules_local [ 228.411924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 228.567932] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 228.584874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 228.587093] Lustre: Skipped 2 previous similar messages [ 229.443787] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 232.126640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 233.279667] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 236.178080] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 236.184910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 237.855082] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 240.674040] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 240.680957] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 240.702693] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 240.707667] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 240.752498] Lustre: lustre-OST0001: new disk, initializing [ 240.755723] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 241.866137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 241.868896] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 241.900280] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 242.307733] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 246.491444] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 247.501302] Lustre: Setting parameter general.lod.*.mdt_hash in log params [ 257.269388] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 257.272089] LustreError: Skipped 1 previous similar message [ 257.273315] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 257.277962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 262.125013] Lustre: server umount lustre-MDT0000 complete [ 262.918936] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 262.928721] LustreError: Skipped 3 previous similar messages [ 263.493982] LustreError: 25416:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709053958 with bad export cookie 13699269355392087151 [ 263.494736] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 263.504922] LustreError: 25416:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 263.619028] Lustre: server umount lustre-MDT0001 complete [ 275.028678] Lustre: 30063:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1709053960/real 1709053960] req@ffff88008d351c00 x1792072720925120/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1709053970 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 275.049717] Lustre: server umount lustre-OST0000 complete [ 286.088632] Lustre: 30336:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1709053971/real 1709053971] req@ffff8800a19d3480 x1792072720925504/t0(0) o39->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1709053981 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 286.139142] Lustre: server umount lustre-OST0001 complete [ 289.235249] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing load_modules_local [ 291.662126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 291.797688] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 292.626003] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 295.120555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 295.253171] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 295.256117] Lustre: Skipped 4 previous similar messages [ 296.195571] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 298.589447] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 298.593682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 299.950044] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 301.995984] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 301.998906] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 303.231200] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 309.595697] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 310.628066] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 310.630765] Lustre: Skipped 1 previous similar message [ 320.276640] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 320.279681] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 320.284234] Lustre: Skipped 3 previous similar messages [ 322.907929] Lustre: server umount lustre-MDT0000 complete [ 323.881384] LustreError: 32599:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709054019 with bad export cookie 13699269355392088397 [ 323.882647] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 323.890857] LustreError: 32599:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 323.980612] Lustre: server umount lustre-MDT0001 complete [ 334.978675] Lustre: 3661:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1709054020/real 1709054020] req@ffff8800a6b03480 x1792072720943872/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1709054030 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 335.002133] Lustre: server umount lustre-OST0000 complete [ 346.014680] Lustre: 3946:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1709054031/real 1709054031] req@ffff8800a6b00000 x1792072720944384/t0(0) o39->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1709054041 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 346.072652] Lustre: server umount lustre-OST0001 complete [ 349.667824] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 12:14:05 (1709054045) [ 356.223263] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing load_modules_local [ 358.552546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 358.674410] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 358.680393] LustreError: Skipped 1 previous similar message [ 359.489461] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 361.906006] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 362.824737] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 365.040498] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 365.044259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 366.344188] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 368.450542] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 368.453291] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 369.710937] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 375.890496] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 376.755223] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 386.980948] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 386.981330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 386.981331] Lustre: Skipped 8 previous similar messages [ 386.987366] Lustre: Skipped 6 previous similar messages [ 388.741042] Lustre: server umount lustre-MDT0000 complete [ 389.684264] LustreError: 10339:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709054084 with bad export cookie 13699269355392089468 [ 389.685609] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 389.691070] LustreError: 10339:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 389.775708] Lustre: server umount lustre-MDT0001 complete [ 400.756854] Lustre: 11081:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1709054085/real 1709054085] req@ffff88007f7d3b80 x1792072720962240/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1709054095 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 400.784484] Lustre: server umount lustre-OST0000 complete [ 412.037943] Lustre: server umount lustre-OST0001 complete [ 422.129325] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 12:15:17 (1709054117) [ 429.283343] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing load_modules_local [ 431.886150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 432.055621] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 432.062642] LustreError: Skipped 1 previous similar message [ 432.083996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 432.086805] Lustre: Skipped 6 previous similar messages [ 433.336073] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 437.187972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 438.645833] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 441.806684] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 441.813481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 443.725590] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 446.881972] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 446.889374] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 448.681895] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 451.496288] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 465.381382] Lustre: server umount lustre-MDT0000 complete [ 467.246730] LustreError: 17019:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709054162 with bad export cookie 13699269355392090539 [ 467.247605] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 467.261453] LustreError: 17019:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 473.468443] Lustre: server umount lustre-MDT0001 complete [ 477.009436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 478.064246] Lustre: Evicted from MGS (at 192.168.204.153@tcp) after server handle changed from 0x0 to 0xbe1d94e59a8a460b [ 478.068733] Lustre: MGC192.168.204.153@tcp: Connection restored to 192.168.204.153@tcp (at 0@lo) [ 479.669322] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 483.699640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 485.052673] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 486.728076] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 491.935025] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 192.168.204.153@tcp (at 0@lo) [ 500.486108] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 13 sec [ 501.142448] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 501.592601] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 501.942361] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.204.153@tcp (at 0@lo) [ 501.946816] Lustre: Skipped 1 previous similar message [ 503.909004] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 504.777692] Lustre: server umount lustre-MDT0000 complete [ 506.103726] LustreError: 13378:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709054201 with bad export cookie 13699269355392091659 [ 506.105066] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 506.110721] LustreError: 13378:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 1 previous similar message [ 506.948972] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 506.955751] LustreError: Skipped 12 previous similar messages [ 508.258776] Lustre: server umount lustre-MDT0001 complete [ 509.890125] Lustre: server umount lustre-OST0000 complete [ 511.163502] Lustre: server umount lustre-OST0001 complete [ 515.484902] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 12:16:51 (1709054211) [ 518.573537] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing load_modules_local [ 521.672372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 523.112205] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 526.223613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 527.537653] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 531.538904] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 531.547538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 534.975160] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 539.096241] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 539.101639] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 541.871291] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 547.356071] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 549.327466] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 549.329612] Lustre: Skipped 2 previous similar messages [ 561.429210] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 561.432925] LustreError: Skipped 1 previous similar message [ 561.435386] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 561.445520] Lustre: Skipped 11 previous similar messages [ 561.448487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 561.453980] Lustre: Skipped 16 previous similar messages [ 566.035778] Lustre: server umount lustre-MDT0000 complete [ 567.787804] Lustre: server umount lustre-MDT0001 complete [ 579.648894] Lustre: 26982:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1709054264/real 1709054264] req@ffff88012fe6f480 x1792072721019904/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1709054274 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 579.662949] Lustre: 26982:0:(client.c:2338:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 579.736092] Lustre: server umount lustre-OST0000 complete [ 592.256346] Lustre: server umount lustre-OST0001 complete [ 600.203194] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 12:18:15 (1709054295) [ 601.433921] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 606.144412] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 12:18:21 (1709054301) [ 610.900896] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_hostid [ 614.258396] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing load_modules_local [ 618.407933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 620.737779] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 620.751588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 623.797108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 623.824110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 623.965148] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 624.003798] Lustre: lustre-MDT0000: new disk, initializing [ 624.068999] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 625.556807] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 629.134314] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 632.614184] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 632.618689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 632.652990] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 632.659643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 634.432057] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 634.453145] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 635.311577] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 638.651680] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 639.792045] Lustre: DEBUG MARKER: create test files [ 656.425402] Lustre: server umount lustre-OST0000 complete [ 658.590366] Lustre: server umount lustre-MDT0000 complete [ 663.073246] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing unload_modules_local [ 664.167168] Key type lgssc unregistered [ 664.253370] LNet: 1109:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 664.259147] LNet: Removed LNI 192.168.204.153@tcp [ 674.545247] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 674.593995] alg: No test for adler32 (adler32-zlib) [ 675.416175] Lustre: Lustre: Build Version: 2.15.61_76_g2ee32a4 [ 675.492758] LNet: Added LNI 192.168.204.153@tcp [8/256/0/180] [ 675.495122] LNet: Accept secure, port 988 [ 677.030962] Key type lgssc registered [ 677.340698] Lustre: Echo OBD driver; http://www.lustre.org/ [ 680.307293] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 680.324006] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 681.760839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 683.417120] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 685.754967] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 688.813785] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 688.823324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 688.963688] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 690.961262] Lustre: lustre-OST0000: deleting orphan objects from 0x240000400:102 to 0x240000400:129 [ 691.007179] Lustre: 4437:0:(osd_internal.h:1363:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 514 < left 544, rollback = 9 [ 691.011316] Lustre: 4437:0:(osd_handler.c:1972:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 691.015419] Lustre: 4437:0:(osd_handler.c:1979:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 691.020207] Lustre: 4437:0:(osd_handler.c:1989:osd_trans_dump_creds()) write: 2/13/1, punch: 0/0/0, quota 1/3/0 [ 691.023931] Lustre: 4437:0:(osd_handler.c:1996:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 691.026555] Lustre: 4437:0:(osd_handler.c:2003:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 691.053352] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 692.641400] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 695.523218] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 698.598102] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 705.989180] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 705.989349] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 705.989859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 707.214537] Lustre: server umount lustre-OST0000 complete [ 709.303004] Lustre: server umount lustre-MDT0000 complete [ 714.507671] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing unload_modules_local [ 715.718494] Key type lgssc unregistered [ 715.814850] LNet: 5734:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 715.817812] LNet: Removed LNI 192.168.204.153@tcp [ 729.543693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 729.595657] alg: No test for adler32 (adler32-zlib) [ 730.450509] Lustre: Lustre: Build Version: 2.15.61_76_g2ee32a4 [ 730.537350] LNet: Added LNI 192.168.204.153@tcp [8/256/0/180] [ 730.540265] LNet: Accept secure, port 988 [ 732.086741] Key type lgssc registered [ 732.505271] Lustre: Echo OBD driver; http://www.lustre.org/ [ 735.999638] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 736.015729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 737.312302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 738.688751] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 740.444165] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 743.057025] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 743.063202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 743.200988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 743.229723] mount.lustre (8823) used greatest stack depth: 9472 bytes left [ 745.503859] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 747.207358] Lustre: lustre-OST0000: deleting orphan objects from 0x240000400:140 to 0x240000400:161 [ 747.264139] Lustre: 8830:0:(osd_internal.h:1363:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 514 < left 544, rollback = 9 [ 747.273037] Lustre: 8830:0:(osd_handler.c:1972:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 747.276378] Lustre: 8830:0:(osd_handler.c:1979:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 747.279581] Lustre: 8830:0:(osd_handler.c:1989:osd_trans_dump_creds()) write: 2/13/1, punch: 0/0/0, quota 1/3/0 [ 747.282379] Lustre: 8830:0:(osd_handler.c:1996:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 747.285395] Lustre: 8830:0:(osd_handler.c:2003:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 747.545616] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 750.404000] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 755.487339] Lustre: server umount lustre-OST0000 complete [ 767.399671] Lustre: 9878:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1709054452/real 1709054452] req@ffff880095709500 x1792073452753216/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1709054462 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 767.580050] Lustre: server umount lustre-MDT0000 complete [ 771.477983] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing unload_modules_local [ 772.226060] Key type lgssc unregistered [ 772.304398] LNet: 10412:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 772.308612] LNet: Removed LNI 192.168.204.153@tcp [ 783.955168] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 783.984647] alg: No test for adler32 (adler32-zlib) [ 784.821802] Lustre: Lustre: Build Version: 2.15.61_76_g2ee32a4 [ 784.881540] LNet: Added LNI 192.168.204.153@tcp [8/256/0/180] [ 784.883500] LNet: Accept secure, port 988 [ 786.422712] Key type lgssc registered [ 786.798840] Lustre: Echo OBD driver; http://www.lustre.org/ [ 790.706678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 793.949065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 796.695821] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 796.714695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 799.994654] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 800.002658] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 803.079881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 803.116802] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 803.126661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 804.290649] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 804.314150] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 804.366617] Lustre: lustre-MDT0000: new disk, initializing [ 804.425820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 804.436035] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 805.837502] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 810.594162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 810.619326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 810.665568] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 810.687188] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 810.690624] Lustre: Skipped 1 previous similar message [ 810.742127] Lustre: lustre-MDT0001: new disk, initializing [ 810.778873] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 810.792714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 810.798640] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 812.265397] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 815.380824] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 816.309952] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 818.847590] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 818.854868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 818.886847] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 818.891685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 819.004458] Lustre: lustre-OST0000: new disk, initializing [ 819.009210] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 819.054936] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 819.791044] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 819.795289] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 819.828115] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 820.840845] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 824.139998] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 825.732131] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 825.823801] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 827.129723] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 827.249288] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 829.828392] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 829.828642] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 829.828989] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 829.840776] LustreError: Skipped 1 previous similar message [ 834.154632] Lustre: server umount lustre-OST0000 complete [ 839.844305] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 839.849480] Lustre: Skipped 1 previous similar message [ 839.852356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 839.856083] Lustre: Skipped 2 previous similar messages [ 841.982843] Lustre: server umount lustre-MDT0000 complete [ 843.711348] LustreError: 14123:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709054538 with bad export cookie 5729800765724750952 [ 843.713401] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 843.723655] LustreError: 14123:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 843.926284] Lustre: server umount lustre-MDT0001 complete [ 850.754146] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 12:22:26 (1709054546) [ 854.627752] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing load_modules_local [ 859.739658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 870.728496] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_hostid [ 873.108689] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing load_modules_local [ 877.395757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 879.978072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 882.240917] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 882.248594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 884.293560] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 884.299114] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 886.619322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 886.652655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 886.749533] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 886.762878] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 886.798045] Lustre: lustre-MDT0000: new disk, initializing [ 886.838390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 886.845635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 887.873120] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 892.211255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 892.236912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 892.268449] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 892.286132] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 892.289127] Lustre: Skipped 1 previous similar message [ 892.325804] Lustre: lustre-MDT0001: new disk, initializing [ 892.352479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 892.367104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 892.371766] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 893.351310] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 896.096024] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 896.959409] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 899.471474] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 899.478210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 899.501067] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 899.507332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 901.472032] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 904.380112] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 904.540778] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 904.544747] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 904.587115] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 905.575021] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 905.651329] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 906.788383] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 906.860753] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 909.588175] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 909.589276] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 909.589277] Lustre: Skipped 1 previous similar message [ 909.589749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 909.603386] LustreError: Skipped 1 previous similar message [ 913.581604] Lustre: server umount lustre-OST0000 complete [ 917.380908] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 917.384385] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 917.389986] Lustre: Skipped 1 previous similar message [ 917.393739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 917.395790] Lustre: Skipped 1 previous similar message [ 920.908366] Lustre: server umount lustre-MDT0000 complete [ 922.281911] LustreError: 24510:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709054617 with bad export cookie 5729800765724751736 [ 922.283636] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 922.290615] LustreError: 24510:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 922.431618] Lustre: server umount lustre-MDT0001 complete [ 926.967797] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 12:23:42 (1709054622) [ 929.048546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 929.243251] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 929.270833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 929.273845] Lustre: Skipped 1 previous similar message [ 930.210475] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 932.898693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 934.222867] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 935.512704] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 936.313314] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 938.542062] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 938.551375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 940.387046] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 942.032411] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 943.541804] Lustre: 31361:0:(genops.c:1641:obd_export_evict_by_uuid()) MGS: evicting 90390ed0-34ad-4bf9-8b72-34d2ec384d3d at adminstrative request [ 943.674702] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 943.683641] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 953.686601] Lustre: MGS: Client 90390ed0-34ad-4bf9-8b72-34d2ec384d3d (at 0@lo) reconnecting [ 953.690619] Lustre: Evicted from MGS (at 192.168.204.153@tcp) after server handle changed from 0x4f8457704ac8fac0 to 0x4f8457704ac8fd52 [ 953.698026] Lustre: MGC192.168.204.153@tcp: Connection restored to 192.168.204.153@tcp (at 0@lo) [ 976.558897] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 976.574602] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 976.606988] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 976.615829] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 976.734883] Lustre: lustre-OST0001: new disk, initializing [ 976.736965] Lustre: Skipped 1 previous similar message [ 976.739632] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 976.743488] Lustre: Skipped 1 previous similar message [ 976.802966] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 976.805697] Lustre: Skipped 2 previous similar messages [ 979.186031] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 982.710196] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 983.890002] Lustre: server umount lustre-OST0001 complete [ 984.772720] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 988.757709] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 988.764238] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 988.770913] Lustre: Skipped 1 previous similar message [ 988.775410] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 988.778493] Lustre: Skipped 2 previous similar messages [ 989.797043] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 989.806333] LustreError: Skipped 3 previous similar messages [ 991.698639] Lustre: server umount lustre-OST0000 complete [ 999.759206] Lustre: server umount lustre-MDT0000 complete [ 999.814225] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 999.822030] LustreError: Skipped 1 previous similar message [ 1001.626115] LustreError: 28669:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709054696 with bad export cookie 5729800765724753234 [ 1001.628221] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1001.656852] LustreError: 28669:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1001.856796] Lustre: server umount lustre-MDT0001 complete [ 1010.527434] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 12:25:05 (1709054705) [ 1011.331024] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 1015.403911] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 12:25:10 (1709054710) [ 1016.317913] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 1020.482292] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 12:25:16 (1709054716) [ 1021.387232] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 1025.749399] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 12:25:21 (1709054721) [ 1026.556468] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 1028.870186] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1028.882494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1029.479618] loop: module loaded [ 1029.495217] LDISKFS-fs (loop0): file extents enabled, maximum tree depth=5 [ 1029.502304] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 1030.333671] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 1031.221637] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 1032.745130] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 1035.573023] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1036.855215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1038.136547] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1038.143260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1039.501608] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1039.506756] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1041.693633] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1041.699777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1041.730605] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1041.736935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1041.748382] LustreError: 5995:0:(obd_config.c:781:class_setup()) setup lustre-OST0000-osd failed (-28) [ 1041.751946] LustreError: 5995:0:(obd_mount.c:213:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 1041.754500] LustreError: 5995:0:(tgt_mount.c:2190:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 1041.758739] LustreError: 5995:0:(super25.c:188:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 1047.223217] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_hostid [ 1049.215100] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing load_modules_local [ 1051.877311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1053.919988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1055.838918] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1055.845170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1058.098921] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1058.105776] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1060.546123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1060.578207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1060.680391] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1060.692068] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1060.731140] Lustre: lustre-MDT0000: new disk, initializing [ 1060.762973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1060.770196] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1061.821717] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 1066.351296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1066.375278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1066.534822] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1067.689679] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 1070.718568] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1071.615872] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1074.156474] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1074.162377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1074.194284] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1074.203079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1075.899344] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 1078.684295] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1079.680329] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1080.654278] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1080.690801] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1080.772259] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 1081.955879] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1082.050859] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1085.684544] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1085.685166] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1085.685168] Lustre: Skipped 3 previous similar messages [ 1085.685606] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1085.685608] Lustre: Skipped 5 previous similar messages [ 1085.698053] LustreError: Skipped 2 previous similar messages [ 1088.932124] Lustre: server umount lustre-OST0000 complete [ 1096.228877] Lustre: server umount lustre-MDT0000 complete [ 1097.413507] LustreError: 9504:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709054792 with bad export cookie 5729800765724753801 [ 1097.416425] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1097.423888] LustreError: 9504:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1097.554044] Lustre: server umount lustre-MDT0001 complete [ 1102.147937] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 12:26:37 (1709054797) [ 1104.473423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1104.644718] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1105.687162] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 1108.456758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1109.670370] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 1111.247169] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1112.083404] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1114.625235] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1114.631795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1116.789416] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 1118.659745] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1120.022481] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1130.460739] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec [ 1132.171987] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1132.322548] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1135.307252] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1135.313724] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1135.354650] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1135.360403] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1135.425222] Lustre: lustre-OST0001: new disk, initializing [ 1135.426877] Lustre: Skipped 2 previous similar messages [ 1135.428660] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 1135.430771] Lustre: Skipped 3 previous similar messages [ 1135.536419] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1135.539896] Lustre: Skipped 5 previous similar messages [ 1137.545099] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 1140.432212] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 1141.726786] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 1144.778310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1144.782084] Lustre: Skipped 2 previous similar messages [ 1144.783742] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1144.825692] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1144.917390] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 1146.305016] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 1146.389845] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1157.042865] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1169.765101] Lustre: Failing over lustre-MDT0000 [ 1169.846022] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1169.847891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1169.847892] Lustre: Skipped 5 previous similar messages [ 1169.854827] Lustre: Skipped 6 previous similar messages [ 1169.983257] Lustre: server umount lustre-MDT0000 complete [ 1172.301369] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.204.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1172.305091] LustreError: Skipped 1 previous similar message [ 1185.876791] LDISKFS-fs (dm-0): recovery complete [ 1185.879182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1185.936017] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1186.114814] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1187.301729] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 1187.301831] Lustre: 18803:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 1187.565923] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 1191.110394] Lustre: 15516:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 1191.111483] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.204.153@tcp (at 0@lo) [ 1191.117927] Lustre: 15516:0:(ldlm_lib.c:2001:extend_recovery_timer()) Skipped 1 previous similar message [ 1191.146065] LustreError: 21344:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 1191.452217] LustreError: 21344:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 1191.857901] LustreError: 21344:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 1191.862994] LustreError: 21344:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 1 previous similar message [ 1192.172635] LustreError: 21344:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 1192.176006] LustreError: 21344:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 1 previous similar message [ 1192.183972] Lustre: 21344:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1192.188748] Lustre: 21344:0:(ldlm_lib.c:2001:extend_recovery_timer()) Skipped 7 previous similar messages [ 1192.896842] LustreError: 21344:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 1192.899509] LustreError: 21344:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 2 previous similar messages [ 1193.207845] LustreError: 21344:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 1193.210877] LustreError: 21344:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 2 previous similar messages [ 1194.530567] Lustre: 21344:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1194.537938] Lustre: 21344:0:(ldlm_lib.c:2001:extend_recovery_timer()) Skipped 6 previous similar messages [ 1195.153030] LustreError: 21344:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 1195.155923] LustreError: 21344:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 6 previous similar messages [ 1195.460465] LustreError: 21344:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 1195.463812] LustreError: 21344:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 6 previous similar messages [ 1198.801659] Lustre: 21344:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1198.807712] Lustre: 21344:0:(ldlm_lib.c:2001:extend_recovery_timer()) Skipped 11 previous similar messages [ 1199.214283] LustreError: 21344:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 1199.221043] LustreError: 21344:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 10 previous similar messages [ 1199.527330] LustreError: 21344:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 1199.530243] LustreError: 21344:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 10 previous similar messages [ 1206.980947] Lustre: 21344:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1206.987349] Lustre: 21344:0:(ldlm_lib.c:2001:extend_recovery_timer()) Skipped 21 previous similar messages [ 1207.390302] LustreError: 21344:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 1207.396839] LustreError: 21344:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 21 previous similar messages [ 1207.699788] LustreError: 21344:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 1207.702896] LustreError: 21344:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 21 previous similar messages [ 1223.020989] Lustre: 21344:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1223.027222] Lustre: 21344:0:(ldlm_lib.c:2001:extend_recovery_timer()) Skipped 43 previous similar messages [ 1223.732037] LustreError: 21344:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 1223.735562] LustreError: 21344:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 43 previous similar messages [ 1223.747990] LustreError: 21344:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 1223.755681] LustreError: 21344:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 44 previous similar messages [ 1247.006034] Lustre: 21344:0:(ldlm_lib.c:2309:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 1247.010102] Lustre: 21344:0:(ldlm_lib.c:2319:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1247.015903] Lustre: 21344:0:(genops.c:1498:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 2f8c7a9a-2b15-4b8e-aaf9-5e76b890d3a3@192.168.204.53@tcp [ 1247.029832] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 1247.033585] Lustre: 21344:0:(ldlm_lib.c:1833:abort_req_replay_queue()) @@@ aborted: req@ffff8800a528e450 x1792073506256192/t0(8589934750) o35->2f8c7a9a-2b15-4b8e-aaf9-5e76b890d3a3@192.168.204.53@tcp:712/0 lens 392/0 e 0 to 0 dl 1709054952 ref 1 fl Complete:/204/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 1247.054145] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 1247.059781] Lustre: lustre-MDT0000: Denying connection for new client 2f8c7a9a-2b15-4b8e-aaf9-5e76b890d3a3 (at 192.168.204.53@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) already passed deadline 0:00 [ 1247.082755] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 1247.103748] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 192.168.204.153@tcp (at 0@lo) [ 1247.107069] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 1247.116157] Lustre: Skipped 2 previous similar messages [ 1247.130056] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000401:40 to 0x2c0000401:545 [ 1247.131200] Lustre: lustre-OST0000: deleting orphan objects from 0x280000401:73 to 0x280000401:577 [ 1256.232317] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1256.234394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1256.234396] Lustre: Skipped 3 previous similar messages [ 1256.247295] Lustre: Skipped 1 previous similar message [ 1260.919352] Lustre: server umount lustre-OST0000 complete [ 1261.237395] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1261.246402] LustreError: Skipped 17 previous similar messages [ 1262.389759] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1268.436848] Lustre: server umount lustre-OST0001 complete [ 1275.925763] Lustre: server umount lustre-MDT0000 complete [ 1277.757570] LustreError: 18137:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709054972 with bad export cookie 5729800765724804957 [ 1277.763442] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1277.779878] LustreError: 18137:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1277.969349] Lustre: server umount lustre-MDT0001 complete [ 1285.139466] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 12:29:40 (1709054980) [ 1289.082370] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1289.094871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1289.099139] Lustre: *** cfs_fail_loc=197, val=0*** [ 1306.226984] LustreError: 24285:0:(mgc_request.c:251:do_config_log_add()) MGC192.168.204.153@tcp: failed processing log, type 1: rc = -5 [ 1322.245784] LustreError: 24285:0:(mgc_request.c:251:do_config_log_add()) MGC192.168.204.153@tcp: failed processing log, type 4: rc = -110 [ 1448.281586] Lustre: *** cfs_fail_loc=197, val=0*** [ 1448.282833] Lustre: Skipped 65 previous similar messages [ 1471.321268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1471.326554] Lustre: Skipped 1 previous similar message [ 1473.627797] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 1475.856127] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1476.844028] Lustre: server umount lustre-OST0000 complete [ 1483.569112] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 12:32:59 (1709055179) [ 1498.549112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1498.823948] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1498.835107] LustreError: Skipped 4 previous similar messages [ 1500.619222] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 1504.132738] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1506.352677] LustreError: 26668:0:(osp_object.c:637:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1506.359785] LustreError: 26668:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1506.363373] LustreError: 26668:0:(lod_dev.c:525:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 1506.488914] Lustre: server umount lustre-MDT0000 complete [ 1511.543695] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1511.549696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1517.391117] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_hostid [ 1520.121083] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing load_modules_local [ 1523.377380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1525.757334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1527.850427] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1527.856838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1529.969988] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1529.975320] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1532.433309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1532.470969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1532.555046] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1532.558249] Lustre: Skipped 1 previous similar message [ 1532.572417] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1532.612425] Lustre: lustre-MDT0000: new disk, initializing [ 1532.677208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1533.851904] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 1538.514684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1538.561988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1538.711359] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1539.992693] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 1543.253383] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1544.192745] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1546.815474] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1546.822620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1546.843707] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1546.849709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1548.686370] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 1550.398365] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1551.540855] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1552.547315] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1552.613881] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1553.608644] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1553.672056] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1555.380391] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1555.380437] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1555.380439] Lustre: Skipped 4 previous similar messages [ 1555.381164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1555.381165] Lustre: Skipped 7 previous similar messages [ 1555.398178] LustreError: Skipped 1 previous similar message [ 1560.406418] Lustre: server umount lustre-OST0000 complete [ 1567.985729] Lustre: server umount lustre-MDT0000 complete [ 1569.999940] LustreError: 31623:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709055265 with bad export cookie 5729800765724815324 [ 1570.002327] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1570.002751] LustreError: 11924:0:(client.c:1281:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88009013f100 x1792073510557696/t0(0) o41->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 224/224 e 0 to 0 dl 1709055274 ref 1 fl Rpc:RESQU/200/0 rc -5/-107 job:'' uid:0 gid:0 [ 1570.015440] LustreError: 31623:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1570.188032] Lustre: server umount lustre-MDT0001 complete [ 1576.900251] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 12:34:32 (1709055272) [ 1580.735078] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing unload_modules_local [ 1581.602917] Key type lgssc unregistered [ 1581.717865] LNet: 4240:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1581.729853] LNet: Removed LNI 192.168.204.153@tcp [ 1595.992853] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1596.093967] alg: No test for adler32 (adler32-zlib) [ 1596.984380] Lustre: Lustre: Build Version: 2.15.61_76_g2ee32a4 [ 1597.066838] LNet: Added LNI 192.168.204.153@tcp [8/256/0/180] [ 1597.069927] LNet: Accept secure, port 988 [ 1598.610320] Key type lgssc registered [ 1599.076617] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1603.125437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1606.141095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1608.679124] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1608.691095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1611.772945] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1611.782394] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1614.514883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1617.980723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1618.001326] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1618.009073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1619.160025] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1619.181075] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1619.246106] Lustre: lustre-MDT0000: new disk, initializing [ 1619.320646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1619.338221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1621.012060] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 1627.222933] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1627.234585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1627.265873] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1627.272711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1627.391661] Lustre: lustre-OST0000: new disk, initializing [ 1627.394832] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1627.396996] Lustre: Skipped 1 previous similar message [ 1627.436830] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1627.599992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1627.607370] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1627.651388] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1630.209935] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 1635.833896] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1635.860485] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1635.915735] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1635.928191] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1635.997695] Lustre: lustre-OST0001: new disk, initializing [ 1635.999504] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 1636.073742] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1638.503952] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 1646.085725] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1646.086491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1646.092749] Lustre: Skipped 1 previous similar message [ 1648.719973] Lustre: server umount lustre-MDT0000 complete [ 1651.113660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1652.767966] LustreError: 8240:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709055347 with bad export cookie 7045148170330549351 [ 1652.769584] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1652.778139] LustreError: 8240:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1652.788382] Lustre: server umount lustre-OST0000 complete [ 1654.105038] Lustre: server umount lustre-OST0001 complete [ 1659.434939] Lustre: DEBUG MARKER: == conf-sanity test 88: check the default mount options can be overridden ========================================================== 12:35:55 (1709055355) [ 1661.614379] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing load_modules_local [ 1664.571752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1664.939024] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1667.021319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1667.351402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1671.501204] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 12:36:07 (1709055367) [ 1675.823710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1676.489444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1677.243224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1678.064849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1678.490713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1679.303793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1685.043539] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_hostid [ 1687.117952] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing load_modules_local [ 1690.596921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1693.142321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1695.282357] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1695.287632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1697.568350] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1697.577151] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1700.314478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1700.347832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1700.443738] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1700.461388] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1700.505013] Lustre: lustre-MDT0000: new disk, initializing [ 1700.551948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1700.562244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1701.760992] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 1706.030664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1706.053774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1706.088464] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1706.112594] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1706.115432] Lustre: Skipped 1 previous similar message [ 1706.149240] Lustre: lustre-MDT0001: new disk, initializing [ 1706.184714] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1706.199000] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1706.203050] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1707.146584] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 1709.692317] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1710.463116] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1712.647202] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1712.653138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1712.677154] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1712.683021] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1714.726634] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 1717.764728] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1718.104122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1718.107674] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1718.141959] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1718.949833] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1719.015687] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1720.274663] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1720.343096] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1723.124823] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1723.131340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1723.133168] Lustre: Skipped 2 previous similar messages [ 1727.220199] Lustre: server umount lustre-OST0000 complete [ 1731.220236] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1731.223887] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1731.229024] Lustre: Skipped 1 previous similar message [ 1731.231173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1734.552334] Lustre: server umount lustre-MDT0000 complete [ 1735.749485] LustreError: 19420:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709055430 with bad export cookie 7045148170330550457 [ 1735.750983] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1735.756890] LustreError: 19420:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1735.875829] Lustre: server umount lustre-MDT0001 complete [ 1740.316779] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 12:37:16 (1709055436) [ 1742.795220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1742.992489] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1743.020508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1743.023013] Lustre: Skipped 1 previous similar message [ 1744.058934] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 1746.886025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1747.963430] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 1749.268814] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1750.012469] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1752.307605] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1752.312548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1754.212340] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 1755.915990] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1761.742758] Lustre: *** cfs_fail_loc=159, val=0*** [ 1777.763464] Lustre: lustre-MDT0000: Client 5f08998c-4656-4aeb-bfb3-22b9595a13dd (at 192.168.204.53@tcp) reconnecting [ 1778.075154] Lustre: *** cfs_fail_loc=159, val=0*** [ 1778.076708] Lustre: Skipped 6 previous similar messages [ 1794.076734] Lustre: lustre-MDT0000: Client 5f08998c-4656-4aeb-bfb3-22b9595a13dd (at 192.168.204.53@tcp) reconnecting [ 1797.509236] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1797.509396] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1797.509732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1797.509734] Lustre: Skipped 2 previous similar messages [ 1797.522722] Lustre: Skipped 2 previous similar messages [ 1801.018871] Lustre: server umount lustre-OST0000 complete [ 1802.516234] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1802.523494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1802.526197] Lustre: Skipped 1 previous similar message [ 1807.108370] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1808.416669] Lustre: server umount lustre-MDT0000 complete [ 1809.607735] LustreError: 24536:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709055504 with bad export cookie 7045148170330551297 [ 1809.609348] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1809.621088] LustreError: 24536:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1809.717662] Lustre: server umount lustre-MDT0001 complete [ 1813.272004] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing unload_modules_local [ 1813.928981] Key type lgssc unregistered [ 1814.021715] LNet: 28462:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1814.027267] LNet: Removed LNI 192.168.204.153@tcp [ 1824.722011] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1824.757153] alg: No test for adler32 (adler32-zlib) [ 1825.592458] Lustre: Lustre: Build Version: 2.15.61_76_g2ee32a4 [ 1825.649435] LNet: Added LNI 192.168.204.153@tcp [8/256/0/180] [ 1825.651848] LNet: Accept secure, port 988 [ 1827.195712] Key type lgssc registered [ 1827.593381] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1830.409848] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1830.420812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1831.657482] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1831.686189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1832.819105] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 1835.958369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1836.107487] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1837.256028] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 1838.637119] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1839.541401] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1841.949546] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1841.954073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1842.090011] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1844.049645] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 1845.524805] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1856.095066] Lustre: lustre-OST0000: deleting orphan objects from 0x280000401:42 to 0x280000401:65 [ 1865.277090] Lustre: *** cfs_fail_loc=159, val=0*** [ 1881.297835] Lustre: lustre-MDT0000: Client d02dc9ac-a6fd-4b4b-84c7-d49301e99c28 (at 192.168.204.53@tcp) reconnecting [ 1882.049737] Lustre: *** cfs_fail_loc=159, val=0*** [ 1882.058571] Lustre: Skipped 1 previous similar message [ 1898.048665] Lustre: lustre-MDT0001: Client d02dc9ac-a6fd-4b4b-84c7-d49301e99c28 (at 192.168.204.53@tcp) reconnecting [ 1898.401906] Lustre: *** cfs_fail_loc=159, val=0*** [ 1898.405872] Lustre: Skipped 3 previous similar messages [ 1914.415505] Lustre: lustre-MDT0001: Client d02dc9ac-a6fd-4b4b-84c7-d49301e99c28 (at 192.168.204.53@tcp) reconnecting [ 1915.684286] Lustre: 1049:0:(mdt_handler.c:84:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 1916.845412] Lustre: *** cfs_fail_loc=159, val=0*** [ 1916.850083] Lustre: Skipped 2 previous similar messages [ 1932.852133] Lustre: lustre-MDT0000: Client 5d4d035e-16fe-4c2f-8687-d037057ffdc1 (at 192.168.204.53@tcp) reconnecting [ 1933.268456] Lustre: *** cfs_fail_loc=159, val=0*** [ 1933.271759] Lustre: Skipped 13 previous similar messages [ 1949.286020] Lustre: lustre-MDT0000: Client 5d4d035e-16fe-4c2f-8687-d037057ffdc1 (at 192.168.204.53@tcp) reconnecting [ 1949.639670] Lustre: 1399:0:(mdt_handler.c:84:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 1951.253753] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1951.254100] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1951.254251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1951.265116] Lustre: Skipped 1 previous similar message [ 1956.260814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1956.263709] Lustre: Skipped 2 previous similar messages [ 1957.082277] Lustre: server umount lustre-OST0000 complete [ 1961.269068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1961.277347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1964.562396] Lustre: server umount lustre-MDT0000 complete [ 1965.850045] LustreError: 30340:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709055661 with bad export cookie 13560997947921641547 [ 1965.852474] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1965.862119] LustreError: 30340:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1965.984307] Lustre: server umount lustre-MDT0001 complete [ 1969.708353] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing unload_modules_local [ 1970.462089] Key type lgssc unregistered [ 1970.538963] LNet: 2600:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1970.542007] LNet: Removed LNI 192.168.204.153@tcp [ 1980.833248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1980.865480] alg: No test for adler32 (adler32-zlib) [ 1981.702966] Lustre: Lustre: Build Version: 2.15.61_76_g2ee32a4 [ 1981.765828] LNet: Added LNI 192.168.204.153@tcp [8/256/0/180] [ 1981.771255] LNet: Accept secure, port 988 [ 1983.308929] Key type lgssc registered [ 1983.559093] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1985.711770] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1985.720887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1986.878986] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1986.902103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1987.864921] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 1990.576717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1990.709358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1991.669279] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 1993.047831] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1993.810960] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1995.941144] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1995.945784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1996.069616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1997.071495] Lustre: lustre-OST0000: deleting orphan objects from 0x280000401:85 to 0x280000401:129 [ 1997.546211] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 1998.833425] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2000.074111] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:97 [ 2005.250334] Lustre: 7070:0:(mdt_handler.c:84:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 2005.900360] Lustre: 7135:0:(mdt_handler.c:84:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 2006.365162] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight in log params [ 2007.518073] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight in log params [ 2007.521167] Lustre: Skipped 1 previous similar message [ 2008.128880] Lustre: 7472:0:(mdt_handler.c:84:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 2010.084924] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2010.090710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2010.092041] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2010.097049] Lustre: Skipped 1 previous similar message [ 2014.999991] Lustre: server umount lustre-OST0000 complete [ 2020.100585] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2020.101145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2020.109704] Lustre: Skipped 2 previous similar messages [ 2022.508302] Lustre: server umount lustre-MDT0000 complete [ 2024.157425] LustreError: 6361:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709055719 with bad export cookie 16834786988584658407 [ 2024.158397] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2024.191729] LustreError: 6361:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2024.431888] Lustre: server umount lustre-MDT0001 complete [ 2028.551036] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing unload_modules_local [ 2029.265948] Key type lgssc unregistered [ 2029.358832] LNet: 8672:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2029.362295] LNet: Removed LNI 192.168.204.153@tcp [ 2039.554646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2039.603239] alg: No test for adler32 (adler32-zlib) [ 2040.434493] Lustre: Lustre: Build Version: 2.15.61_76_g2ee32a4 [ 2040.506543] LNet: Added LNI 192.168.204.153@tcp [8/256/0/180] [ 2040.509274] LNet: Accept secure, port 988 [ 2042.053698] Key type lgssc registered [ 2042.393950] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2045.039837] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2045.052278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2046.296291] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2046.322076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2047.413004] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 2050.935124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2051.135752] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2052.665916] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 2054.282401] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2055.208008] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2057.881700] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2057.888988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2058.069135] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2060.089820] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 2061.070500] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:129 [ 2061.539986] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2071.073243] Lustre: lustre-OST0000: deleting orphan objects from 0x280000401:85 to 0x280000401:161 [ 2073.697511] Lustre: *** cfs_fail_loc=159, val=0*** [ 2089.707197] Lustre: lustre-MDT0000: Client 292aae36-83ae-4ac9-8250-aad4f7b418dc (at 192.168.204.53@tcp) reconnecting [ 2091.108297] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2091.108746] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2091.109047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2091.116409] LustreError: Skipped 1 previous similar message [ 2096.118288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2096.121140] Lustre: Skipped 2 previous similar messages [ 2096.737908] Lustre: server umount lustre-OST0000 complete [ 2101.126356] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2101.126958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2101.134341] Lustre: Skipped 2 previous similar messages [ 2104.129692] Lustre: server umount lustre-MDT0000 complete [ 2105.528769] LustreError: 10552:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709055800 with bad export cookie 376036181291053887 [ 2105.530837] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2105.536777] LustreError: 10552:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2105.654332] Lustre: server umount lustre-MDT0001 complete [ 2108.966461] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing unload_modules_local [ 2109.593483] Key type lgssc unregistered [ 2109.676172] LNet: 14346:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2109.679057] LNet: Removed LNI 192.168.204.153@tcp [ 2120.143070] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2120.174125] alg: No test for adler32 (adler32-zlib) [ 2121.026807] Lustre: Lustre: Build Version: 2.15.61_76_g2ee32a4 [ 2121.121370] LNet: Added LNI 192.168.204.153@tcp [8/256/0/180] [ 2121.136596] LNet: Accept secure, port 988 [ 2122.707758] Key type lgssc registered [ 2123.054139] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2126.138903] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2126.147815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2127.392246] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2127.420212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2128.852998] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 2132.357381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2132.494363] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2133.501157] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 2134.916803] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2135.750706] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2138.119583] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2138.125955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2138.252210] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2139.935019] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 2141.499234] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2142.256535] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:161 [ 2152.258031] Lustre: lustre-OST0000: deleting orphan objects from 0x280000401:170 to 0x280000401:193 [ 2169.005531] Lustre: lustre-OST0000: evicting 3c3a1f6b-f6ff-438d-b859-9e0efa2e8f3d (at 192.168.204.53@tcp) by administrative request [ 2169.009151] Lustre: Skipped 1 previous similar message [ 2197.332954] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2197.333150] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2197.333383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2197.343794] Lustre: Skipped 1 previous similar message [ 2198.929880] Lustre: server umount lustre-OST0000 complete [ 2202.340802] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2202.341915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2202.341916] Lustre: Skipped 1 previous similar message [ 2202.347270] Lustre: Skipped 1 previous similar message [ 2206.387381] Lustre: server umount lustre-MDT0000 complete [ 2207.348278] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2207.353292] LustreError: Skipped 1 previous similar message [ 2207.521025] LustreError: 16222:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709055902 with bad export cookie 9621389397177846276 [ 2207.521851] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2207.530484] LustreError: 16222:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2207.633611] Lustre: server umount lustre-MDT0001 complete [ 2210.958439] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing unload_modules_local [ 2211.555776] Key type lgssc unregistered [ 2211.624290] LNet: 20551:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2211.629326] LNet: Removed LNI 192.168.204.153@tcp [ 2254.628221] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2254.658645] alg: No test for adler32 (adler32-zlib) [ 2255.484555] Lustre: Lustre: Build Version: 2.15.61_76_g2ee32a4 [ 2255.538227] LNet: Added LNI 192.168.204.153@tcp [8/256/0/180] [ 2255.540192] LNet: Accept secure, port 988 [ 2257.077016] Key type lgssc registered [ 2257.372947] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2260.281737] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2260.300951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2261.468013] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2261.488814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2262.605845] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 2265.513199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2265.664661] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2266.684324] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 2268.341859] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2269.211055] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2271.750371] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2271.756458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2271.906751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2273.840779] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 2274.913138] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:193 [ 2275.721282] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2285.912324] Lustre: lustre-OST0000: deleting orphan objects from 0x280000401:195 to 0x280000401:225 [ 2289.949350] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2289.953764] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2289.970643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2290.916952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2290.917586] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2290.918059] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2290.935895] Lustre: Skipped 1 previous similar message [ 2294.984913] Lustre: server umount lustre-OST0000 complete [ 2300.708395] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2300.711381] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2300.717462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2302.554211] Lustre: server umount lustre-MDT0000 complete [ 2303.809483] LustreError: 26197:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709055998 with bad export cookie 1441321686036106560 [ 2303.811472] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2303.817933] LustreError: 26197:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2303.935484] Lustre: server umount lustre-MDT0001 complete [ 2308.025070] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing unload_modules_local [ 2309.025884] Key type lgssc unregistered [ 2309.106332] LNet: 29854:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2309.109600] LNet: Removed LNI 192.168.204.153@tcp [ 2320.006580] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2320.029949] alg: No test for adler32 (adler32-zlib) [ 2320.916076] Lustre: Lustre: Build Version: 2.15.61_76_g2ee32a4 [ 2321.042101] LNet: Added LNI 192.168.204.153@tcp [8/256/0/180] [ 2321.047429] LNet: Accept secure, port 988 [ 2322.660974] Key type lgssc registered [ 2323.070743] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2325.437257] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2325.443673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2329.899202] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 12:47:05 (1709056025) [ 2335.617007] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_hostid [ 2338.242682] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing load_modules_local [ 2341.636911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2344.183687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2346.719771] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2346.727907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2349.375726] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2349.383507] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2351.946422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2351.978314] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2351.988145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2353.086731] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2353.106753] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2353.140952] Lustre: lustre-MDT0000: new disk, initializing [ 2353.174470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2353.214429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2354.511837] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 2359.505855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2359.544906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2359.612220] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2359.640228] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2359.642983] Lustre: Skipped 1 previous similar message [ 2359.679565] Lustre: lustre-MDT0001: new disk, initializing [ 2359.708915] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2359.722674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2359.728392] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2360.981398] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 2363.986072] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2364.820556] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2367.286869] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2367.294287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2367.318884] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2367.324983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2367.414284] Lustre: lustre-OST0000: new disk, initializing [ 2367.416803] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2367.450086] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2369.212088] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 2372.038096] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2373.451989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2373.456053] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2373.484981] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2382.487201] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2382.490964] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2382.497954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2383.492287] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2383.496075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2383.499962] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2386.126375] Lustre: server umount lustre-OST0000 complete [ 2388.501588] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2388.501595] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2388.502359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2388.502360] Lustre: Skipped 1 previous similar message [ 2393.446597] Lustre: server umount lustre-MDT0000 complete [ 2393.508251] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2393.513872] LustreError: Skipped 1 previous similar message [ 2394.577466] LustreError: 6490:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709056089 with bad export cookie 14006226124373118834 [ 2394.578890] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2394.589453] LustreError: 6490:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2394.694770] Lustre: server umount lustre-MDT0001 complete [ 2398.312890] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing unload_modules_local [ 2399.073300] Key type lgssc unregistered [ 2399.164186] LNet: 7286:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2399.167488] LNet: Removed LNI 192.168.204.153@tcp [ 2410.357027] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2410.395861] alg: No test for adler32 (adler32-zlib) [ 2411.219835] Lustre: Lustre: Build Version: 2.15.61_76_g2ee32a4 [ 2411.282393] LNet: Added LNI 192.168.204.153@tcp [8/256/0/180] [ 2411.285342] LNet: Accept secure, port 988 [ 2412.838697] Key type lgssc registered [ 2413.126375] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2415.737964] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2415.748170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2416.918722] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2416.935844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2418.103885] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 2421.814698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2421.995373] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2423.207327] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 2424.876667] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2425.836192] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2428.667549] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2428.673660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2428.826061] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2430.514353] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 2431.898377] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2441.831323] Lustre: lustre-OST0000: deleting orphan objects from 0x280000401:3 to 0x280000401:33 [ 2444.624133] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2444.642687] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2444.654729] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2444.658156] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2444.670742] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.153@tcp (at 0@lo) [ 2445.514497] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2445.518521] Lustre: Skipped 1 previous similar message [ 2445.524660] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2445.529542] Lustre: Skipped 1 previous similar message [ 2445.531730] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2445.534607] Lustre: Skipped 1 previous similar message [ 2445.538035] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2445.542596] LustreError: Skipped 1 previous similar message [ 2445.554242] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.153@tcp (at 0@lo) [ 2445.558780] Lustre: Skipped 1 previous similar message [ 2446.863412] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2446.865393] Lustre: Skipped 2 previous similar messages [ 2446.876833] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2446.883038] Lustre: Skipped 2 previous similar messages [ 2446.885637] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2446.888442] Lustre: Skipped 2 previous similar messages [ 2446.890985] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2446.896051] LustreError: Skipped 2 previous similar messages [ 2446.904134] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.153@tcp (at 0@lo) [ 2446.906044] Lustre: Skipped 2 previous similar messages [ 2449.016694] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2449.018885] Lustre: Skipped 4 previous similar messages [ 2449.024889] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2449.029270] Lustre: Skipped 4 previous similar messages [ 2449.031076] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2449.034138] Lustre: Skipped 4 previous similar messages [ 2449.035796] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2449.040009] LustreError: Skipped 4 previous similar messages [ 2449.042381] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.153@tcp (at 0@lo) [ 2449.045598] Lustre: Skipped 4 previous similar messages [ 2453.307998] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2453.310941] Lustre: Skipped 10 previous similar messages [ 2453.319046] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2453.322709] Lustre: Skipped 10 previous similar messages [ 2453.326671] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2453.331578] Lustre: Skipped 10 previous similar messages [ 2453.333692] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2453.338957] LustreError: Skipped 10 previous similar messages [ 2453.342773] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.153@tcp (at 0@lo) [ 2453.345832] Lustre: Skipped 10 previous similar messages [ 2461.772928] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2461.775687] Lustre: Skipped 18 previous similar messages [ 2461.784555] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2461.788614] Lustre: Skipped 18 previous similar messages [ 2461.792826] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2461.795772] Lustre: Skipped 18 previous similar messages [ 2461.797812] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2461.802581] LustreError: Skipped 18 previous similar messages [ 2461.807808] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.153@tcp (at 0@lo) [ 2461.810840] Lustre: Skipped 18 previous similar messages [ 2478.149226] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2478.151225] Lustre: Skipped 36 previous similar messages [ 2478.167009] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2478.171045] Lustre: Skipped 36 previous similar messages [ 2478.172604] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2478.174639] Lustre: Skipped 36 previous similar messages [ 2478.176293] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2478.179481] LustreError: Skipped 36 previous similar messages [ 2478.182401] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.153@tcp (at 0@lo) [ 2478.184571] Lustre: Skipped 36 previous similar messages [ 2510.323405] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2510.329088] Lustre: Skipped 80 previous similar messages [ 2510.336256] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2510.342967] Lustre: Skipped 80 previous similar messages [ 2510.345339] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2510.348039] Lustre: Skipped 80 previous similar messages [ 2510.349814] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2510.356091] LustreError: Skipped 80 previous similar messages [ 2510.359374] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.153@tcp (at 0@lo) [ 2510.368041] Lustre: Skipped 80 previous similar messages [ 2574.441457] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2574.444091] Lustre: Skipped 153 previous similar messages [ 2574.455327] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2574.466625] Lustre: Skipped 153 previous similar messages [ 2574.469044] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2574.472555] Lustre: Skipped 153 previous similar messages [ 2574.475734] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2574.479844] LustreError: Skipped 153 previous similar messages [ 2574.484013] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.153@tcp (at 0@lo) [ 2574.491501] Lustre: Skipped 153 previous similar messages [ 2702.457249] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2702.459278] Lustre: Skipped 323 previous similar messages [ 2702.464982] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2702.469763] Lustre: Skipped 323 previous similar messages [ 2702.471790] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2702.473798] Lustre: Skipped 323 previous similar messages [ 2702.873069] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2702.876618] LustreError: Skipped 324 previous similar messages [ 2702.878256] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.153@tcp (at 0@lo) [ 2702.880742] Lustre: Skipped 324 previous similar messages [ 3831.606859] LustreError: 11024:0:(ofd_dev.c:1691:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 3831.611825] LustreError: 9202:0:(osp_precreate.c:670:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 3903.169940] hrtimer: interrupt took 3289832 ns [ 5060.388827] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5060.395180] Lustre: Skipped 363 previous similar messages [ 5060.397223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5060.400201] Lustre: Skipped 1 previous similar message [ 5065.027195] Lustre: server umount lustre-OST0000 complete [ 5070.406142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5070.412159] Lustre: Skipped 1 previous similar message [ 5073.236686] Lustre: server umount lustre-MDT0000 complete [ 5075.231762] LustreError: 9167:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709058770 with bad export cookie 9864897025654719405 [ 5075.234451] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5075.243907] LustreError: 9167:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5075.437450] Lustre: server umount lustre-MDT0001 complete [ 5080.069917] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing unload_modules_local [ 5081.195081] Key type lgssc unregistered [ 5081.351239] LNet: 17838:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5081.359759] LNet: Removed LNI 192.168.204.153@tcp [ 5094.789296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5094.853080] alg: No test for adler32 (adler32-zlib) [ 5095.704023] Lustre: Lustre: Build Version: 2.15.61_76_g2ee32a4 [ 5095.775211] LNet: Added LNI 192.168.204.153@tcp [8/256/0/180] [ 5095.777467] LNet: Accept secure, port 988 [ 5097.319697] Key type lgssc registered [ 5097.650184] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5100.561067] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5100.571406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5101.793202] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5101.826791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5103.025157] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 5106.514907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5106.670506] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5107.911394] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 5110.145253] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5111.389506] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5114.902455] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5114.912268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5115.098224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5117.500977] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 5119.141889] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5126.101647] Lustre: lustre-OST0000: deleting orphan objects from 0x280000401:49666 to 0x280000401:49697 [ 5131.223326] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5131.981337] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5131.988750] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5131.994472] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5132.000840] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.153@tcp (at 0@lo) [ 5132.002288] LustreError: 19752:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 5134.365924] Lustre: lustre-OST0000: Not available for connect from 192.168.204.53@tcp (stopping) [ 5136.100723] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5136.102476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5136.110357] Lustre: Skipped 1 previous similar message [ 5138.897850] Lustre: server umount lustre-OST0000 complete [ 5152.076649] LustreError: 19752:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 2107 awake [ 5168.371181] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5168.380015] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5168.568201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5171.390324] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 5173.670261] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5175.600051] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 5176.566742] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5176.575120] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.153@tcp (at 0@lo) [ 5176.580146] Lustre: Skipped 1 previous similar message [ 5177.205234] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff88012bfa8800.ost_server_uuid 50 [ 5177.968536] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012bfa8800.ost_server_uuid in FULL state after 0 sec [ 5181.583612] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5181.587363] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5181.588992] Lustre: Skipped 1 previous similar message [ 5185.722791] Lustre: server umount lustre-OST0000 complete [ 5191.588642] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5191.591077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5191.594675] Lustre: Skipped 2 previous similar messages [ 5193.616034] Lustre: server umount lustre-MDT0000 complete [ 5195.664447] LustreError: 19717:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709058890 with bad export cookie 2476931518686238100 [ 5195.678735] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5195.882790] Lustre: server umount lustre-MDT0001 complete [ 5201.094333] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing unload_modules_local [ 5202.255325] Key type lgssc unregistered [ 5202.360071] LNet: 25109:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5202.368066] LNet: Removed LNI 192.168.204.153@tcp [ 5220.692273] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5220.753698] alg: No test for adler32 (adler32-zlib) [ 5221.619871] Lustre: Lustre: Build Version: 2.15.61_76_g2ee32a4 [ 5221.745032] LNet: Added LNI 192.168.204.153@tcp [8/256/0/180] [ 5221.749219] LNet: Accept secure, port 988 [ 5223.371689] Key type lgssc registered [ 5223.824286] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5228.861123] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5228.883778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5230.167177] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5230.187741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5231.475356] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 5235.512818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5235.724160] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5237.139921] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 5241.170587] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5241.183149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5241.395053] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5243.221596] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 5245.402954] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:67 to 0x280000400:97 [ 5247.089767] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 5247.104978] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5247.135977] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 5247.142268] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5247.216463] Lustre: lustre-OST0001: new disk, initializing [ 5247.218556] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 5247.268750] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5249.724851] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 5252.541935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5252.549138] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5252.603374] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5255.586510] Lustre: lustre-OST0000: deleting orphan objects from 0x280000401:49666 to 0x280000401:49729 [ 5256.270874] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5261.031510] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 5278.506415] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 5278.512499] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5278.517921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5280.628852] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5280.629382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5280.637501] Lustre: Skipped 1 previous similar message [ 5281.914285] Lustre: server umount lustre-MDT0000 complete [ 5283.619654] LustreError: 30120:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709058978 with bad export cookie 11522922728721938639 [ 5283.624676] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5283.635269] LustreError: 30120:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5283.904200] Lustre: server umount lustre-MDT0001 complete [ 5295.655101] Lustre: 32032:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1709058980/real 1709058980] req@ffff88008fc65c00 x1792078161974208/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1709058990 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 5295.698469] Lustre: server umount lustre-OST0000 complete [ 5307.440930] Lustre: 32306:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1709058992/real 1709058992] req@ffff88009c885880 x1792078161974592/t0(0) o39->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1709059002 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 5307.557666] Lustre: server umount lustre-OST0001 complete [ 5308.943799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 5308.995634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5309.504173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 5309.541027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5309.975137] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5309.983191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 5310.021663] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5310.028774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5310.410258] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 5310.416445] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 5310.454395] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 5310.465439] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5315.364944] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing load_modules_local [ 5319.883095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5320.126179] LustreError: 137-5: mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5320.209511] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5321.851143] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 5325.173577] LustreError: 137-5: mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5325.969448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5327.697429] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 5331.698215] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5331.718637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5331.900660] Lustre: mylustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5331.904655] Lustre: Skipped 1 previous similar message [ 5334.530807] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 5335.926743] LustreError: 137-5: mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5335.936841] Lustre: mylustre-OST0000: deleting orphan objects from 0x280000400:67 to 0x280000400:129 [ 5338.556558] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 5338.563229] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5340.647417] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 5345.095718] Lustre: mylustre-OST0001: deleting orphan objects from 0x2c0000401:3 to 0x2c0000401:33 [ 5345.097791] Lustre: mylustre-OST0000: deleting orphan objects from 0x280000401:49731 to 0x280000401:49761 [ 5349.432648] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5350.997900] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 5363.615660] Lustre: server umount mylustre-MDT0000 complete [ 5365.147345] Lustre: mylustre-MDT0000-lwp-OST0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5365.153795] Lustre: Skipped 1 previous similar message [ 5365.154919] LustreError: 137-5: mylustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5365.604914] LustreError: 1535:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709059060 with bad export cookie 11522922728721941663 [ 5365.606693] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5365.611783] LustreError: 1535:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5365.815363] Lustre: server umount mylustre-MDT0001 complete [ 5377.900686] Lustre: 6062:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1709059063/real 1709059063] req@ffff8800a1d75880 x1792078162000448/t0(0) o39->mylustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1709059073 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 5377.944328] Lustre: server umount mylustre-OST0000 complete [ 5389.644608] Lustre: 6339:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1709059074/real 1709059074] req@ffff88012e288a80 x1792078162000832/t0(0) o39->mylustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1709059084 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 5389.747642] Lustre: server umount mylustre-OST0001 complete [ 5391.132447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 5391.169839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5391.809706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 5391.845388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5392.340325] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5392.350811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 5392.403793] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5392.409582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5393.104574] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 5393.116860] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 5393.147120] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 5393.155104] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5399.995214] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing load_modules_local [ 5405.446463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5405.646196] LustreError: 137-5: tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5405.652957] LustreError: Skipped 3 previous similar messages [ 5405.681967] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5405.685848] Lustre: Skipped 1 previous similar message [ 5407.090148] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 5411.116438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5413.229474] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 5418.031453] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5418.040736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5421.229451] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 5421.309277] Lustre: tfs-OST0000: deleting orphan objects from 0x280000400:67 to 0x280000400:161 [ 5426.038382] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 5426.048020] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5428.334414] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 5430.246213] Lustre: tfs-OST0000: deleting orphan objects from 0x280000401:49731 to 0x280000401:49793 [ 5430.246486] Lustre: tfs-OST0001: deleting orphan objects from 0x2c0000401:35 to 0x2c0000401:65 [ 5437.096141] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5438.668732] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 5450.278721] Lustre: tfs-MDT0000-lwp-OST0000: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5450.280225] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 5450.280227] Lustre: Skipped 3 previous similar messages [ 5450.295365] Lustre: Skipped 1 previous similar message [ 5452.606866] Lustre: server umount tfs-MDT0000 complete [ 5454.246752] LustreError: 8008:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709059149 with bad export cookie 11522922728721943742 [ 5454.247430] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5454.273035] LustreError: 8008:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5454.458023] Lustre: server umount tfs-MDT0001 complete [ 5466.318699] Lustre: 12450:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1709059151/real 1709059151] req@ffff88007341b100 x1792078162022272/t0(0) o39->tfs-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1709059161 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 5466.373661] Lustre: server umount tfs-OST0000 complete [ 5478.447698] Lustre: 12723:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1709059163/real 1709059163] req@ffff88007341b100 x1792078162022656/t0(0) o39->tfs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1709059173 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 5478.588583] Lustre: server umount tfs-OST0001 complete [ 5480.291201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 5480.350150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5481.042338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 5481.079756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5481.630962] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5481.639948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 5481.687961] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5481.695021] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5482.158991] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 5482.164929] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 5482.191616] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 5482.198427] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5487.496961] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing load_modules_local [ 5490.852690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5491.112579] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5491.120556] LustreError: Skipped 2 previous similar messages [ 5491.152932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5491.156994] Lustre: Skipped 3 previous similar messages [ 5492.538420] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 5497.208133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5499.525610] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 5504.058790] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5504.068433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5507.166621] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 5507.314179] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:67 to 0x280000400:193 [ 5511.712131] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 5511.730721] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5514.607232] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 5516.666686] Lustre: lustre-OST0000: deleting orphan objects from 0x280000401:49731 to 0x280000401:49825 [ 5516.667867] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000401:35 to 0x2c0000401:97 [ 5523.318730] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5524.952631] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 5536.870002] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 13:40:32 (1709059232) [ 5551.716763] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5551.717341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5551.717342] Lustre: Skipped 3 previous similar messages [ 5551.726185] Lustre: Skipped 5 previous similar messages [ 5554.820995] Lustre: server umount lustre-MDT0000 complete [ 5556.652365] LustreError: 15287:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709059251 with bad export cookie 11522922728721945597 [ 5556.654965] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5556.665067] LustreError: 15287:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5556.724476] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5556.726264] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5556.726266] Lustre: Skipped 3 previous similar messages [ 5556.734613] LustreError: Skipped 3 previous similar messages [ 5562.874884] Lustre: server umount lustre-MDT0001 complete [ 5570.783871] Lustre: server umount lustre-OST0000 complete [ 5578.676199] Lustre: server umount lustre-OST0001 complete [ 5581.056902] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_hostid [ 5583.707782] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing load_modules_local [ 5587.632269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5590.810599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5593.799105] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5593.815392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5597.225342] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 5597.235340] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5600.928672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5601.022977] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 5601.038348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 5601.188415] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5601.212997] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5601.270901] Lustre: lustre-MDT0000: new disk, initializing [ 5601.331907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5601.336253] Lustre: Skipped 3 previous similar messages [ 5601.361883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5602.976276] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 5609.260386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5609.299687] LDISKFS-fs (dm-1): Mount option "noacl" will be removed by 3.5 Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 5609.323612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 5609.385465] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5609.424409] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5609.426720] Lustre: Skipped 1 previous similar message [ 5609.479040] Lustre: lustre-MDT0001: new disk, initializing [ 5609.585338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5609.591567] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5611.519922] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 5618.254897] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5618.260840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5618.294278] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5618.298870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5618.406845] Lustre: lustre-OST0000: new disk, initializing [ 5618.408871] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5620.317143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5620.320000] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5620.366175] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5621.164419] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 5627.042603] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 5627.048765] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5627.104166] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 5627.112635] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5627.209848] Lustre: lustre-OST0001: new disk, initializing [ 5627.213716] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 5628.404349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5628.412411] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5628.457866] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5630.181960] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 5639.608992] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 13:42:15 (1709059335) [ 5648.645279] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 13:42:24 (1709059344) [ 5663.493817] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5663.495183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5663.495185] Lustre: Skipped 3 previous similar messages [ 5663.505102] Lustre: Skipped 3 previous similar messages [ 5666.731852] Lustre: server umount lustre-OST0000 complete [ 5668.503544] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5668.507688] LustreError: Skipped 2 previous similar messages [ 5669.652328] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5675.232355] Lustre: server umount lustre-MDT0000 complete [ 5677.203460] LustreError: 22285:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709059372 with bad export cookie 11522922728721947340 [ 5677.206062] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5677.209465] Lustre: lustre-OST0001: haven't heard from client ea966f38-e34d-4e23-a1af-a40cc6780236 (at 192.168.204.53@tcp) in 31 seconds. I think it's dead, and I am evicting it. exp ffff88008850f000, cur 1709059372 expire 1709059342 last 1709059341 [ 5677.227203] LustreError: 22285:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5682.534267] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5682.537216] Lustre: Skipped 6 previous similar messages [ 5683.435685] Lustre: server umount lustre-MDT0001 complete [ 5689.290542] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing unload_modules_local [ 5695.687954] Lustre: server umount lustre-OST0001 complete [ 5699.079218] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_hostid [ 5707.551107] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing load_modules_local [ 5714.896794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5719.175166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5723.036625] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5723.053037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5727.108021] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 5727.116893] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5731.472183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5731.509861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5731.683246] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5731.723341] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5731.803400] Lustre: lustre-MDT0000: new disk, initializing [ 5731.883504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5731.886189] Lustre: Skipped 3 previous similar messages [ 5731.896893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5733.804243] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 5740.390981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5740.419885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5740.732311] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5742.311890] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 5746.392707] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5747.708789] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5751.516949] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5751.531059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5751.573991] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5751.580342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5751.685253] Lustre: lustre-OST0000: new disk, initializing [ 5751.687472] Lustre: Skipped 1 previous similar message [ 5751.693437] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5751.699196] Lustre: Skipped 2 previous similar messages [ 5752.852565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5752.856029] Lustre: Skipped 1 previous similar message [ 5752.858071] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5752.914238] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5754.699652] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 5758.510576] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5762.885639] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5762.889635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5762.898044] Lustre: Skipped 5 previous similar messages [ 5767.436213] Lustre: server umount lustre-OST0000 complete [ 5770.662804] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5775.098776] Lustre: server umount lustre-MDT0000 complete [ 5777.370604] LustreError: 31562:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709059472 with bad export cookie 11522922728721948768 [ 5777.373508] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5777.395297] LustreError: 31562:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5777.656642] Lustre: server umount lustre-MDT0001 complete [ 5784.390059] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing unload_modules_local [ 5786.280022] Key type lgssc unregistered [ 5786.473500] LNet: 3013:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5786.478459] LNet: Removed LNI 192.168.204.153@tcp [ 5805.344946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5805.399466] alg: No test for adler32 (adler32-zlib) [ 5806.332786] Lustre: Lustre: Build Version: 2.15.61_76_g2ee32a4 [ 5806.468539] LNet: Added LNI 192.168.204.153@tcp [8/256/0/180] [ 5806.474395] LNet: Accept secure, port 988 [ 5808.047712] Key type lgssc registered [ 5808.505401] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5812.847905] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5812.858153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5814.220931] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5814.252285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5816.003367] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 5820.398559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5820.610425] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5822.201330] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 5824.211253] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5825.728637] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5829.652834] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5829.660038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5829.872914] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5832.833996] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 5835.331298] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5835.973901] Lustre: Setting parameter lustre-OST0000.ost.unknown_param in log lustre-OST0000 [ 5838.867084] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5838.867930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5843.083748] Lustre: server umount lustre-OST0000 complete [ 5845.620282] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5845.627250] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5845.643981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5845.646745] Lustre: Skipped 2 previous similar messages [ 5848.885430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5848.886111] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5848.893790] Lustre: Skipped 1 previous similar message [ 5851.380043] Lustre: server umount lustre-MDT0000 complete [ 5853.308569] LustreError: 4974:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709059548 with bad export cookie 13750419340984185375 [ 5853.318948] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5853.325885] LustreError: 4974:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5853.587213] Lustre: server umount lustre-MDT0001 complete [ 5857.937231] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing unload_modules_local [ 5859.573039] Key type lgssc unregistered [ 5859.687754] LNet: 8687:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5859.696734] LNet: Removed LNI 192.168.204.153@tcp [ 5868.844095] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5869.159178] alg: No test for adler32 (adler32-zlib) [ 5870.058950] Lustre: Lustre: Build Version: 2.15.61_76_g2ee32a4 [ 5870.146968] LNet: Added LNI 192.168.204.153@tcp [8/256/0/180] [ 5870.151960] LNet: Accept secure, port 988 [ 5871.738306] Key type lgssc registered [ 5872.196237] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5876.860111] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5876.864888] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5876.870679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5894.033363] LustreError: 9913:0:(mgc_request.c:251:do_config_log_add()) MGC192.168.204.153@tcp: failed processing log, type 1: rc = -5 [ 5930.045729] LustreError: 9913:0:(mgc_request.c:251:do_config_log_add()) MGC192.168.204.153@tcp: failed processing log, type 4: rc = -110 [ 6080.114452] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6082.664125] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 6085.114607] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6086.355335] Lustre: server umount lustre-OST0000 complete [ 6093.089772] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing unload_modules_local [ 6094.527258] Key type lgssc unregistered [ 6094.691535] LNet: 11738:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6094.704367] LNet: Removed LNI 192.168.204.153@tcp [ 6122.107398] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6122.167486] alg: No test for adler32 (adler32-zlib) [ 6123.015302] Lustre: Lustre: Build Version: 2.15.61_76_g2ee32a4 [ 6123.081478] LNet: Added LNI 192.168.204.153@tcp [8/256/0/180] [ 6123.083650] LNet: Accept secure, port 988 [ 6124.631223] Key type lgssc registered [ 6125.264978] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6132.727614] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6133.759389] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6134.525005] LDISKFS-fs (loop0): file extents enabled, maximum tree depth=5 [ 6134.534593] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6135.299059] LDISKFS-fs (loop0): file extents enabled, maximum tree depth=5 [ 6135.303845] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6135.779470] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 6137.593814] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 6139.527313] LDISKFS-fs (loop0): file extents enabled, maximum tree depth=5 [ 6139.532295] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 6141.660255] LDISKFS-fs (loop0): file extents enabled, maximum tree depth=5 [ 6141.665795] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 6143.396980] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6143.417714] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6143.426821] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6143.466242] Lustre: 16660:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 6143.471276] Lustre: 16660:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 6143.476749] Lustre: 16660:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 6143.489918] Lustre: 16660:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 6143.493644] Lustre: 16660:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 6143.501722] Lustre: 16660:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 6143.508861] Lustre: 16660:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 6143.513707] Lustre: 16660:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 6143.522248] Lustre: 16660:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 6143.531082] Lustre: 16660:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 6143.537889] Lustre: 16660:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 6143.544951] Lustre: 16660:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 6143.549598] Lustre: 16660:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 6143.553982] Lustre: 16660:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 6147.530348] Lustre: server umount MGS complete [ 6147.977375] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6148.111824] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6148.148965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6148.167810] Lustre: lustre-MDT0000: Aborting client recovery [ 6148.766266] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6148.794805] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6148.821132] Lustre: 17209:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 6148.828060] Lustre: 17209:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 6148.832138] Lustre: 17209:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 6148.837299] Lustre: 17209:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 6148.845840] Lustre: 17209:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 6148.857160] Lustre: 17209:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 6148.862853] Lustre: 17209:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 6148.870973] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6148.995972] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6149.033688] Lustre: lustre-MDT0001: Aborting client recovery [ 6149.569841] LDISKFS-fs (loop2): file extents enabled, maximum tree depth=5 [ 6149.576689] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6149.604442] LDISKFS-fs (loop2): file extents enabled, maximum tree depth=5 [ 6149.614794] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6149.643204] Lustre: 17302:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 6149.649832] Lustre: 17302:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 6149.656585] Lustre: 17302:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 6149.662056] Lustre: 17302:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 6149.665800] Lustre: 17302:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 6149.670772] Lustre: 17302:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 6149.685479] Lustre: 17302:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 6149.769948] Lustre: Found index 0 for lustre-OST0000, updating log [ 6149.831137] Lustre: lustre-OST0000: Aborting recovery [ 6150.434466] LDISKFS-fs (loop3): file extents enabled, maximum tree depth=5 [ 6150.442953] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6150.513043] LDISKFS-fs (loop3): file extents enabled, maximum tree depth=5 [ 6150.520943] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6150.536746] Lustre: 17396:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 6150.543314] Lustre: 17396:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 6150.549452] Lustre: 17396:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 6150.555118] Lustre: 17396:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 6150.560325] Lustre: 17396:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 6150.565521] Lustre: 17396:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 6150.570217] Lustre: 17396:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 6150.610595] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6150.614579] Lustre: Skipped 1 previous similar message [ 6150.639540] Lustre: lustre-OST0001: Aborting recovery [ 6155.634785] LustreError: 17139:0:(lod_dev.c:525:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -108 [ 6158.623568] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6 to 0x0:33 [ 6158.642805] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6 to 0x0:33 [ 6158.668984] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 6159.634883] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:5 to 0x280000400:33 [ 6159.636257] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:5 to 0x2c0000400:33 [ 6161.126378] LustreError: 17109:0:(mdt_identity.c:112:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 455us: rc = -2 [ 6163.641716] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 6174.026576] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6174.032315] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6174.041669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6174.660448] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6174.660750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6174.668697] Lustre: Skipped 1 previous similar message [ 6178.196801] Lustre: server umount lustre-MDT0000 complete [ 6178.551083] LustreError: 17093:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709059873 with bad export cookie 12895580317551064338 [ 6178.554868] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6178.564614] LustreError: 17093:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6179.672368] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6179.672504] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6179.672505] Lustre: Skipped 1 previous similar message [ 6179.672691] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6179.672693] Lustre: Skipped 2 previous similar messages [ 6179.694879] LustreError: Skipped 1 previous similar message [ 6183.686972] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6183.687470] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6183.687472] Lustre: Skipped 1 previous similar message [ 6183.697618] LustreError: Skipped 1 previous similar message [ 6184.779335] Lustre: server umount lustre-MDT0001 complete [ 6191.276033] Lustre: server umount lustre-OST0000 complete [ 6197.976080] Lustre: server umount lustre-OST0001 complete [ 6206.940929] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 13:51:42 (1709059902) [ 6222.525652] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_hostid [ 6225.905479] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing load_modules_local [ 6231.043058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6235.265285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6238.985027] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6239.004167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6242.713260] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 6242.722487] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6247.412491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6247.475579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6247.593063] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6247.609226] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6247.661323] Lustre: lustre-MDT0000: new disk, initializing [ 6247.711056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6247.719396] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6249.324807] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 6255.635235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6255.665104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6255.708695] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6255.737967] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6255.742626] Lustre: Skipped 1 previous similar message [ 6255.794070] Lustre: lustre-MDT0001: new disk, initializing [ 6255.835280] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6255.857111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6255.862838] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6257.469865] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 6261.488308] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6262.921041] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6266.434928] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6266.442310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6266.481853] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6266.487802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6266.612836] Lustre: lustre-OST0000: new disk, initializing [ 6266.615047] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6266.715219] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6269.269943] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 6272.384040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6272.387790] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6272.447860] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6273.553593] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6276.760192] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 6280.000381] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 6285.923998] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb in log lustre-client [ 6332.519949] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6332.520635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6332.520636] Lustre: Skipped 1 previous similar message [ 6332.532567] Lustre: Skipped 2 previous similar messages [ 6334.557197] Lustre: server umount lustre-OST0000 complete [ 6337.524342] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6342.535909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6342.546602] Lustre: Skipped 4 previous similar messages [ 6342.569311] Lustre: server umount lustre-MDT0000 complete [ 6344.518850] LustreError: 22888:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709060039 with bad export cookie 12895580317551067005 [ 6344.526035] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6344.540077] LustreError: 22888:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6344.721981] Lustre: server umount lustre-MDT0001 complete [ 6349.640230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6351.579575] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 6353.944442] Lustre: server umount MGS complete [ 6360.734885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6361.047050] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6361.069413] LustreError: 29630:0:(qmt_pool.c:1518:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 6361.118718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6363.278256] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 6366.118711] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6368.102382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6370.794227] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 6373.501940] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6375.583163] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6379.479940] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6379.487262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6383.049370] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 6386.224736] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6411.733484] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6411.741108] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6411.767369] Lustre: Skipped 1 previous similar message [ 6411.774179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6413.732898] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6417.292025] Lustre: server umount lustre-OST0000 complete [ 6423.588771] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6425.840827] Lustre: server umount lustre-MDT0000 complete [ 6428.634728] LustreError: 29601:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709060123 with bad export cookie 12895580317551068790 [ 6428.638268] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6428.927443] Lustre: server umount lustre-MDT0001 complete [ 6435.297005] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing unload_modules_local [ 6436.613103] Key type lgssc unregistered [ 6436.774858] LNet: 1441:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6436.782099] LNet: Removed LNI 192.168.204.153@tcp [ 6468.873139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6468.928818] alg: No test for adler32 (adler32-zlib) [ 6469.867399] Lustre: Lustre: Build Version: 2.15.61_76_g2ee32a4 [ 6470.003934] LNet: Added LNI 192.168.204.153@tcp [8/256/0/180] [ 6470.007229] LNet: Accept secure, port 988 [ 6471.560445] Key type lgssc registered [ 6472.185693] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6477.010242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6481.381104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6485.233675] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6485.251040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6489.693395] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 6489.730071] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6493.800171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6493.836433] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6493.847630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6495.019392] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6495.037081] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6495.100953] Lustre: lustre-MDT0000: new disk, initializing [ 6495.228067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6495.242377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6497.251980] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 6504.004149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6504.069340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6504.144314] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6504.174064] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6504.176856] Lustre: Skipped 1 previous similar message [ 6504.223651] Lustre: lustre-MDT0001: new disk, initializing [ 6504.313283] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6504.327909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6504.332274] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6506.152949] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 6510.295592] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6511.541403] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6515.261951] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6515.270641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6515.320215] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6515.332335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6515.549638] Lustre: lustre-OST0000: new disk, initializing [ 6515.551686] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6515.602915] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6518.470000] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 6521.837925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6521.842351] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6521.955414] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6522.447828] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6526.235482] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 6531.142236] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 6539.845843] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb in log lustre-client [ 6586.981602] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6586.981968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6586.989860] Lustre: Skipped 1 previous similar message [ 6590.300013] Lustre: server umount lustre-OST0000 complete [ 6594.565900] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6594.569624] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6594.575581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6594.578689] Lustre: Skipped 1 previous similar message [ 6596.998981] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6597.002060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6599.489814] Lustre: server umount lustre-MDT0000 complete [ 6601.763893] LustreError: 6634:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709060296 with bad export cookie 3871843818210808313 [ 6601.766275] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6601.775746] LustreError: 6634:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6601.963219] Lustre: server umount lustre-MDT0001 complete [ 6606.110231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6608.127518] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 6610.254280] Lustre: server umount MGS complete [ 6615.982273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6616.366594] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6616.396336] LustreError: 13584:0:(qmt_pool.c:1518:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 6616.448718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6618.288201] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 6621.446700] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6623.212750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6623.498630] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6625.340914] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 6627.704608] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6629.244148] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6633.309377] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6633.316429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6633.532570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6636.225598] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 6638.611428] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6661.573046] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6661.577978] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6661.579692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6661.579694] Lustre: Skipped 1 previous similar message [ 6661.586610] Lustre: Skipped 1 previous similar message [ 6666.583555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6666.586586] Lustre: Skipped 2 previous similar messages [ 6666.886209] Lustre: server umount lustre-OST0000 complete [ 6671.588479] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6675.094061] Lustre: server umount lustre-MDT0000 complete [ 6676.598063] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6676.602197] LustreError: Skipped 1 previous similar message [ 6676.838349] LustreError: 13554:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709060372 with bad export cookie 3871843818210810126 [ 6676.840184] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6676.855258] LustreError: 13554:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6676.986187] Lustre: server umount lustre-MDT0001 complete [ 6681.673686] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing unload_modules_local [ 6683.084147] Key type lgssc unregistered [ 6683.231658] LNet: 17816:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6683.237806] LNet: Removed LNI 192.168.204.153@tcp [ 6699.838859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6699.891651] alg: No test for adler32 (adler32-zlib) [ 6700.828939] Lustre: Lustre: Build Version: 2.15.61_76_g2ee32a4 [ 6700.968207] LNet: Added LNI 192.168.204.153@tcp [8/256/0/180] [ 6700.973109] LNet: Accept secure, port 988 [ 6702.546705] Key type lgssc registered [ 6702.992176] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6707.542124] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6707.646472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6708.967566] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6709.027557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6710.936870] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 6715.831867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6716.033781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6717.448432] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 6719.279776] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6720.396474] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6723.752177] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6723.765981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6723.999418] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6726.808079] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 6729.458350] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6733.032206] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 6733.038420] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6733.073436] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 6733.077990] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6733.152196] Lustre: lustre-OST0001: new disk, initializing [ 6733.156966] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 6733.202505] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6735.528625] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 6739.002995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6739.006353] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6739.039844] LustreError: 23424:0:(osp_precreate.c:986:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 6740.049637] LustreError: 23424:0:(osp_precreate.c:986:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 6740.062155] LustreError: 23424:0:(osp_precreate.c:986:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [ 6741.058591] LustreError: 23420:0:(osp_precreate.c:986:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 6741.663292] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6741.767068] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6742.986952] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800a8f67000.ost_server_uuid 50 [ 6743.076082] LustreError: 23420:0:(osp_precreate.c:986:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 6743.084857] LustreError: 23420:0:(osp_precreate.c:986:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages [ 6743.694230] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800a8f67000.ost_server_uuid in FULL state after 0 sec [ 6745.281251] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800a8f67000.ost_server_uuid 50 [ 6746.104260] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8800a8f67000.ost_server_uuid in FULL state after 0 sec [ 6747.115275] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6759.063547] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6759.064923] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6759.071857] Lustre: Skipped 1 previous similar message [ 6764.068992] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6764.071040] Lustre: Skipped 2 previous similar messages [ 6764.860990] Lustre: server umount lustre-OST0001 complete [ 6769.076945] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6769.078427] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6769.100010] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6769.103380] Lustre: Skipped 2 previous similar messages [ 6770.256979] Lustre: server umount lustre-OST0000 complete [ 6774.084235] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6774.095541] Lustre: Skipped 1 previous similar message [ 6774.099815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6774.106835] Lustre: Skipped 1 previous similar message [ 6778.810285] LustreError: 24730:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (ffff88008e94cd00) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 6778.841337] Lustre: server umount lustre-MDT0000 complete [ 6779.093154] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6779.098161] LustreError: Skipped 2 previous similar messages [ 6781.146052] LustreError: 19837:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709060476 with bad export cookie 8291517482898551077 [ 6781.151606] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6781.165862] LustreError: 19837:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6781.360249] Lustre: server umount lustre-MDT0001 complete [ 6786.680055] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing unload_modules_local [ 6788.020668] Key type lgssc unregistered [ 6788.200844] LNet: 25529:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6788.206487] LNet: Removed LNI 192.168.204.153@tcp [ 6803.227319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6803.282131] alg: No test for adler32 (adler32-zlib) [ 6804.132613] Lustre: Lustre: Build Version: 2.15.61_76_g2ee32a4 [ 6804.207275] LNet: Added LNI 192.168.204.153@tcp [8/256/0/180] [ 6804.210386] LNet: Accept secure, port 988 [ 6805.745687] Key type lgssc registered [ 6806.178182] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6809.609031] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6809.617403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6810.833618] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6810.906366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6812.516787] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 6814.880893] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6818.488508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6818.711918] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6820.572183] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 6824.823957] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6824.828584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6825.021365] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6827.758259] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 6828.017418] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6829.824392] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6832.693484] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 6832.702025] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6832.871932] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6835.136767] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 6835.873577] Lustre: lustre-OST0000: deleting orphan objects from 0x280000401:68 to 0x280000401:97 [ 6835.882185] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000401:3 to 0x2c0000401:33 [ 6836.905808] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6839.156673] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6839.229811] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6926.042496] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6926.048048] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6926.050742] Lustre: Skipped 1 previous similar message [ 6931.046912] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6931.058555] Lustre: Skipped 1 previous similar message [ 6931.738619] Lustre: server umount lustre-OST0001 complete [ 6936.052962] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6936.053385] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6936.053386] Lustre: Skipped 1 previous similar message [ 6936.053768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6936.085228] LustreError: Skipped 1 previous similar message [ 6940.704023] Lustre: server umount lustre-OST0000 complete [ 6943.908090] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6943.913881] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6943.918675] Lustre: Skipped 1 previous similar message [ 6943.928545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6943.934770] Lustre: Skipped 1 previous similar message [ 6946.070393] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6948.460770] Lustre: server umount lustre-MDT0000 complete [ 6950.598843] LustreError: 31315:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709060645 with bad export cookie 5697808953113268227 [ 6950.605354] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6950.853201] Lustre: server umount lustre-MDT0001 complete [ 6956.721205] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing unload_modules_local [ 6957.935788] Key type lgssc unregistered [ 6958.053193] LNet: 301:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6958.057411] LNet: Removed LNI 192.168.204.153@tcp [ 6984.998834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6985.039954] alg: No test for adler32 (adler32-zlib) [ 6985.951029] Lustre: Lustre: Build Version: 2.15.61_76_g2ee32a4 [ 6986.169778] LNet: Added LNI 192.168.204.153@tcp [8/256/0/180] [ 6986.182441] LNet: Accept secure, port 988 [ 6987.749722] Key type lgssc registered [ 6988.249368] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6993.118032] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6993.135836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6994.376859] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6994.416807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6995.951106] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 7000.925276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7001.316886] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7003.568821] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 7008.294321] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 7008.304296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7008.519968] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7011.338601] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 7011.510649] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7015.840485] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 7015.853285] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7015.985068] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7018.410299] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 7018.987466] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000401:3 to 0x2c0000401:65 [ 7018.989970] Lustre: lustre-OST0000: deleting orphan objects from 0x280000401:68 to 0x280000401:129 [ 7023.293313] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7027.701236] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 7041.285406] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7041.287882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7041.317339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7044.036420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7044.039468] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7044.045448] Lustre: Skipped 3 previous similar messages [ 7045.302083] Lustre: server umount lustre-MDT0000 complete [ 7047.326895] LustreError: 3374:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709060742 with bad export cookie 11120086733724456359 [ 7047.331907] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7047.549137] Lustre: server umount lustre-MDT0001 complete [ 7059.329677] Lustre: 8094:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1709060744/real 1709060744] req@ffff8801315cdf80 x1792080012666304/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1709060754 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 7059.399357] Lustre: server umount lustre-OST0000 complete [ 7071.641670] Lustre: 8368:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1709060756/real 1709060756] req@ffff88009edd4700 x1792080012666688/t0(0) o39->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1709060766 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 7071.950786] Lustre: server umount lustre-OST0001 complete [ 7080.559343] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing load_modules_local [ 7085.997011] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7086.358500] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7086.392167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7088.078282] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 7091.397328] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7092.057676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7093.826007] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 7097.922428] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 7097.937191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7098.145359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7098.148683] Lustre: Skipped 1 previous similar message [ 7101.364821] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 7102.151292] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7105.843614] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 7105.849667] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7108.707943] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 7111.823971] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000401:3 to 0x2c0000401:97 [ 7111.827386] Lustre: lustre-OST0000: deleting orphan objects from 0x280000401:68 to 0x280000401:161 [ 7118.094315] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7119.960874] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 7126.838138] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7126.840600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7126.856509] Lustre: Skipped 5 previous similar messages [ 7131.847119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7131.851420] Lustre: Skipped 6 previous similar messages [ 7132.816310] Lustre: server umount lustre-MDT0000 complete [ 7134.352253] LustreError: 9751:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709060829 with bad export cookie 11120086733724458011 [ 7134.353478] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7134.362492] LustreError: 9751:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7134.525436] Lustre: server umount lustre-MDT0001 complete [ 7146.190674] Lustre: 13989:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1709060831/real 1709060831] req@ffff88009edc2a00 x1792080012687872/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1709060841 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 7146.321351] Lustre: server umount lustre-OST0000 complete [ 7158.151708] Lustre: 14263:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1709060843/real 1709060843] req@ffff88006f5cd500 x1792080012688384/t0(0) o39->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1709060853 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 7158.276951] Lustre: server umount lustre-OST0001 complete [ 7162.134338] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [ 7162.957068] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [ 7166.044433] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 14:07:41 (1709060861) [ 7176.845172] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing load_modules_local [ 7180.787499] SGI XFS with ACLs, security attributes, no debug enabled [ 7180.852764] XFS (loop0): Mounting V5 Filesystem [ 7180.891961] XFS (loop0): Ending clean mount [ 7186.457368] LDISKFS-fs (loop1): file extents enabled, maximum tree depth=5 [ 7186.658607] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7195.998110] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_hostid [ 7200.596105] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing load_modules_local [ 7206.928643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7211.087796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7214.711314] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 7214.730254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7218.601145] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 7218.606828] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7222.436817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7222.468819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7222.599994] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7222.615495] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 7222.711053] Lustre: lustre-MDT0000: new disk, initializing [ 7222.774192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7222.781221] Lustre: Skipped 1 previous similar message [ 7222.790373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7224.331595] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 7230.779213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7230.815282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7230.861309] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7230.886703] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 7230.889216] Lustre: Skipped 1 previous similar message [ 7230.935135] Lustre: lustre-MDT0001: new disk, initializing [ 7231.107613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7231.113956] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7233.152700] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 7236.488984] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7237.992859] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7241.354816] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 7241.362784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7241.401434] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 7241.412316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7241.593509] Lustre: lustre-OST0000: new disk, initializing [ 7241.598812] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 7242.844000] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7242.847382] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7242.909695] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7244.508899] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 7248.736188] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7251.076591] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7251.180683] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7252.870465] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7253.046272] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7257.913139] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7257.922735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7260.200603] Lustre: server umount lustre-OST0000 complete [ 7262.955356] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7262.959885] Lustre: Skipped 1 previous similar message [ 7266.148569] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7266.151465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7266.153260] Lustre: Skipped 2 previous similar messages [ 7269.082828] Lustre: server umount lustre-MDT0000 complete [ 7271.432741] LustreError: 20039:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709060966 with bad export cookie 11120086733724459530 [ 7271.439850] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7271.465468] LustreError: 20039:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7271.736825] Lustre: server umount lustre-MDT0001 complete [ 7276.777712] XFS (loop0): Unmounting Filesystem [ 7283.836052] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 14:09:39 (1709060979) [ 7287.873848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7288.123341] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7288.180844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7288.183445] Lustre: Skipped 2 previous similar messages [ 7289.934145] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 7294.692357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7296.471585] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 7298.966954] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7300.737732] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7304.522228] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 7304.531912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7307.293705] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 7309.867959] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7320.707364] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7320.713457] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7320.732581] Lustre: Skipped 1 previous similar message [ 7320.740593] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7320.743118] Lustre: Skipped 2 previous similar messages [ 7325.376579] Lustre: server umount lustre-OST0000 complete [ 7333.932716] Lustre: server umount lustre-MDT0000 complete [ 7336.233505] LustreError: 28063:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709061031 with bad export cookie 11120086733724460384 [ 7336.237636] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7336.251303] LustreError: 28063:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7336.661496] Lustre: server umount lustre-MDT0001 complete [ 7343.284136] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing unload_modules_local [ 7344.726564] Key type lgssc unregistered [ 7344.865013] LNet: 29136:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7344.874130] LNet: Removed LNI 192.168.204.153@tcp [ 7364.426274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7364.447081] alg: No test for adler32 (adler32-zlib) [ 7365.436415] Lustre: Lustre: Build Version: 2.15.61_76_g2ee32a4 [ 7365.562325] LNet: Added LNI 192.168.204.153@tcp [8/256/0/180] [ 7365.569840] LNet: Accept secure, port 988 [ 7367.185072] Key type lgssc registered [ 7367.749668] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7372.840427] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7372.863381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7374.267679] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7374.315400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7376.048471] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 7380.885856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7381.129843] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7383.374338] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 7385.920082] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7387.087498] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7391.138999] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 7391.151048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7391.235703] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7391.368213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7394.054044] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 7396.334690] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7406.606356] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.53@tcp (stopping) [ 7409.223210] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 7409.227181] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7409.239142] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7411.050113] Lustre: server umount lustre-MDT0001 complete [ 7411.606119] LustreError: 137-5: lustre-MDT0001: not available for connect from 192.168.204.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7412.340189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7414.411827] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7414.423494] LustreError: Skipped 1 previous similar message [ 7416.218676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7416.272628] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7416.274763] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7416.288613] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7416.404966] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7417.883385] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 7421.415688] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 192.168.204.153@tcp (at 0@lo) [ 7421.507914] LustreError: 2686:0:(obd_mount.c:208:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 7421.510875] LustreError: 2686:0:(tgt_mount.c:683:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 7421.518682] LustreError: 31232:0:(mgc_request.c:619:do_requeue()) failed processing log: -17 [ 7421.536827] LustreError: 2689:0:(obd_config.c:2021:class_config_llog_handler()) MGC192.168.204.153@tcp: cfg command failed: rc = -17 [ 7421.540294] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 7726.900797] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7726.909776] Lustre: Skipped 1 previous similar message [ 7726.917687] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7726.920537] Lustre: Skipped 1 previous similar message [ 7728.326888] Lustre: server umount lustre-MDT0001 complete [ 7729.826740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7731.908183] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7731.915660] LustreError: Skipped 1 previous similar message [ 7732.871815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7732.924410] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7732.929070] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7732.947665] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7733.103503] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7734.454461] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 7738.105026] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 192.168.204.153@tcp (at 0@lo) [ 7739.957672] LustreError: 18277:0:(obd_mount.c:208:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 7739.963407] LustreError: 18277:0:(tgt_mount.c:683:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 7739.971011] LustreError: 31232:0:(mgc_request.c:619:do_requeue()) failed processing log: -17 [ 7739.974330] LustreError: 31232:0:(mgc_request.c:619:do_requeue()) Skipped 1 previous similar message [ 7740.005301] LustreError: 18281:0:(obd_config.c:687:class_setup()) Device 8 already setup (type osp) [ 7740.010053] LustreError: 18281:0:(obd_config.c:2021:class_config_llog_handler()) MGC192.168.204.153@tcp: cfg command failed: rc = -17 [ 7740.022401] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.204.153@tcp [ 8037.641400] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.53@tcp (stopping) [ 8037.643209] Lustre: Skipped 1 previous similar message [ 8038.632405] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8042.628876] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.53@tcp (stopping) [ 8042.635339] Lustre: Skipped 2 previous similar messages [ 8043.149518] Lustre: server umount lustre-MDT0001 complete [ 8043.636691] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8043.640907] LustreError: Skipped 2 previous similar messages [ 8044.803083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8048.191179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8048.247231] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8048.250510] Lustre: Found index 1 for lustre-MDT0001, updating log [ 8048.271807] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8048.401921] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 8049.976625] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 8053.416046] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 192.168.204.153@tcp (at 0@lo) [ 8053.485550] LustreError: 1044:0:(obd_mount.c:208:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 8053.488800] LustreError: 1044:0:(tgt_mount.c:683:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 8053.493446] LustreError: 31232:0:(mgc_request.c:619:do_requeue()) failed processing log: -17 [ 8053.498687] LustreError: 31232:0:(mgc_request.c:619:do_requeue()) Skipped 1 previous similar message [ 8053.527041] LustreError: 1047:0:(lod_lov.c:231:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [ 8053.538283] LustreError: 1047:0:(obd_config.c:2021:class_config_llog_handler()) MGC192.168.204.153@tcp: cfg command failed: rc = -114 [ 8053.542722] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [ 8368.966622] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8368.966729] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8368.968008] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8368.980675] Lustre: Skipped 3 previous similar messages [ 8371.441272] Lustre: server umount lustre-MDT0000 complete [ 8373.414545] LustreError: 31221:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709062068 with bad export cookie 808001961136758565 [ 8373.416426] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8373.424716] LustreError: 31221:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8373.972780] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8373.974171] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8373.986301] LustreError: Skipped 1 previous similar message [ 8379.644207] Lustre: server umount lustre-MDT0001 complete [ 8381.891265] Lustre: server umount lustre-OST0000 complete [ 8386.515112] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_hostid [ 8390.242681] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing load_modules_local [ 8395.412075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8398.799312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8402.269683] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 8402.281222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8405.801883] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 8405.810642] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8412.177626] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 14:28:27 (1709062107) [ 8416.145360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8416.173387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8416.299684] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8416.313499] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 8416.376014] Lustre: lustre-MDT0000: new disk, initializing [ 8416.415564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8416.434352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8418.090315] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 8423.558848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8423.587243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8423.633183] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8423.655354] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 8423.658294] Lustre: Skipped 1 previous similar message [ 8423.700025] Lustre: lustre-MDT0001: new disk, initializing [ 8423.863236] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8423.905581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8423.913460] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8425.421162] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 8429.033246] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8430.251160] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8433.877061] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 8433.884030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8433.917889] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 8433.922872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8434.017249] Lustre: lustre-OST0000: new disk, initializing [ 8434.019470] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 8434.057071] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8436.053070] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 8438.026348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8438.033196] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8438.070790] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8439.496734] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8443.060635] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8443.062355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8443.062357] Lustre: Skipped 4 previous similar messages [ 8443.076055] Lustre: Skipped 2 previous similar messages [ 8448.775763] Lustre: server umount lustre-MDT0000 complete [ 8451.459167] LustreError: 19545:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709062146 with bad export cookie 808001961136760462 [ 8451.464656] LustreError: 20511:0:(osp_object.c:637:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [ 8451.464834] LustreError: 20511:0:(llog_cat.c:737:llog_cat_cancel_arr_rec()) lustre-MDT0000-osp-MDT0001: fail to cancel 1 llog-records: rc = -5 [ 8451.464840] LustreError: 20511:0:(llog_cat.c:773:llog_cat_cancel_records()) lustre-MDT0000-osp-MDT0001: fail to cancel 1 of 1 llog-records: rc = -5 [ 8451.465499] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8451.484497] LustreError: 19545:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8451.633730] Lustre: server umount lustre-MDT0001 complete [ 8464.081960] Lustre: 22892:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1709062149/real 1709062149] req@ffff88008d13dc00 x1792080410236096/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1709062159 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 8464.332138] Lustre: server umount lustre-OST0000 complete [ 8474.040386] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 14:29:29 (1709062169) [ 8484.099057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8484.329951] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8484.338259] LustreError: Skipped 1 previous similar message [ 8484.373228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8486.117913] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 8490.628305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8492.420434] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 8494.525186] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8495.890415] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8496.929938] Lustre: Failing over lustre-MDT0000 [ 8497.035846] Lustre: server umount lustre-MDT0000 complete [ 8498.886576] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8498.900339] Lustre: Skipped 1 previous similar message [ 8511.510649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8511.573068] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8511.704822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8511.708026] Lustre: Skipped 1 previous similar message [ 8511.768668] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8513.598775] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 8516.714672] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 8516.717108] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.204.153@tcp (at 0@lo) [ 8516.739934] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 8517.338929] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 8521.716952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8521.720008] Lustre: Skipped 6 previous similar messages [ 8525.826224] Lustre: server umount lustre-MDT0000 complete [ 8527.528559] LustreError: 27954:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709062222 with bad export cookie 808001961136762366 [ 8527.529781] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8527.539841] LustreError: 27954:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8527.786284] Lustre: server umount lustre-MDT0001 complete [ 8531.861242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8532.169353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8533.545686] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 8537.358768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8538.723413] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 8540.519007] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8541.634091] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8542.616788] Lustre: Failing over lustre-MDT0000 [ 8542.710462] Lustre: server umount lustre-MDT0000 complete [ 8547.525998] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8547.529172] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8547.538673] Lustre: Skipped 2 previous similar messages [ 8548.542864] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8548.548204] LustreError: Skipped 16 previous similar messages [ 8556.394967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8556.484950] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8556.494314] Lustre: 28844:0:(mgc_request.c:1825:mgc_process_log()) MGC192.168.204.153@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 8556.606306] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8558.071555] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 8561.357226] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 8561.590410] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 8561.592513] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.204.153@tcp (at 0@lo) [ 8561.592518] Lustre: Skipped 1 previous similar message [ 8561.607133] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 8569.693651] Lustre: server umount lustre-MDT0000 complete [ 8571.574906] LustreError: 28832:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709062266 with bad export cookie 808001961136763262 [ 8571.577883] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8571.587974] LustreError: 28832:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8571.779398] Lustre: server umount lustre-MDT0001 complete [ 8579.284760] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 14:31:14 (1709062274) [ 8587.375867] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_hostid [ 8590.699182] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing load_modules_local [ 8595.582190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8598.621484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8601.146983] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 8601.153260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8603.812254] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 8603.823817] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8607.381630] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing load_modules_local [ 8611.553131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8611.591878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8611.747271] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8611.763733] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 8611.809631] Lustre: lustre-MDT0000: new disk, initializing [ 8611.853936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8611.857789] Lustre: Skipped 2 previous similar messages [ 8611.864899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8613.170737] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 8618.876551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8618.908740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8618.976160] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 8618.978764] Lustre: Skipped 1 previous similar message [ 8619.019782] Lustre: lustre-MDT0001: new disk, initializing [ 8619.092413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8619.100053] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8620.519215] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 8624.132546] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8625.131417] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8627.978490] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 8627.986618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8628.014779] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 8628.024419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8628.122462] Lustre: lustre-OST0000: new disk, initializing [ 8628.128145] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 8629.594871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8629.600322] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8629.626443] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 8629.710183] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8629.739243] LustreError: 5957:0:(ofd_dev.c:1593:ofd_create_hdl()) lustre-OST0000: invalid precreate request for 0x280000401:33, last_id 65281. Likely MDS last_id corruption [ 8629.743114] LustreError: 6191:0:(osp_precreate.c:670:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -22 [ 8629.746101] LustreError: 6191:0:(osp_precreate.c:1372:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -22 [ 8630.655227] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 8634.387176] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8638.166332] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 8639.684341] LustreError: 5957:0:(ofd_dev.c:1593:ofd_create_hdl()) lustre-OST0000: invalid precreate request for 0x280000401:33, last_id 65281. Likely MDS last_id corruption [ 8639.695321] LustreError: 6191:0:(osp_precreate.c:670:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -22 [ 8639.697738] LustreError: 6191:0:(osp_precreate.c:1372:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -22 [ 8649.671320] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8649.674701] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8649.674703] Lustre: Skipped 3 previous similar messages [ 8649.675132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8649.675133] Lustre: Skipped 4 previous similar messages [ 8652.613399] Lustre: server umount lustre-OST0000 complete [ 8659.111302] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8660.373847] Lustre: server umount lustre-MDT0000 complete [ 8662.311462] LustreError: 5972:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709062357 with bad export cookie 808001961136763661 [ 8662.318220] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8662.326546] LustreError: 5972:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8662.521250] Lustre: server umount lustre-MDT0001 complete [ 8667.122800] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing unload_modules_local [ 8667.907490] Key type lgssc unregistered [ 8667.987024] LNet: 8091:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8667.994720] LNet: Removed LNI 192.168.204.153@tcp [ 8680.681296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8680.705967] alg: No test for adler32 (adler32-zlib) [ 8681.561015] Lustre: Lustre: Build Version: 2.15.61_76_g2ee32a4 [ 8681.659154] LNet: Added LNI 192.168.204.153@tcp [8/256/0/180] [ 8681.664332] LNet: Accept secure, port 988 [ 8683.225866] Key type lgssc registered [ 8683.573731] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8686.702717] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8686.713173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8687.907571] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8687.933596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8689.373386] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 8693.217024] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8693.363569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8694.598293] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 8696.245568] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8697.096326] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8699.610717] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 8699.615185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8699.742687] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8701.747157] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 8703.486896] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8703.747669] Lustre: lustre-OST0000: deleting orphan objects from 0x280000402:747 to 0x280000402:833 [ 8727.615690] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 14:33:43 (1709062423) [ 8728.479270] Lustre: Modifying parameter general.jobid_name in log params [ 8729.477835] Lustre: Modifying parameter general.jobid_name in log params [ 8735.310308] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 14:33:50 (1709062430) [ 8741.483998] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 14:33:57 (1709062437) [ 8742.418950] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb in log lustre-client [ 8749.065314] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 14:34:04 (1709062444) [ 8750.179111] Lustre: Setting parameter general.osc.*.max_pages_per_rpc in log params [ 8750.181994] Lustre: Skipped 2 previous similar messages [ 8758.253548] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc in log lustre-client [ 8758.257817] Lustre: Skipped 7 previous similar messages [ 8762.204914] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 14:34:17 (1709062457) [ 8763.486811] Lustre: *** cfs_fail_loc=131b, val=2*** [ 8765.732957] Lustre: *** cfs_fail_loc=131b, val=2*** [ 8772.888943] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 14:34:28 (1709062468) [ 8773.614762] Lustre: Modifying parameter general.jobid_name in log params [ 8773.619387] Lustre: Skipped 3 previous similar messages [ 8781.065596] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 14:34:36 (1709062476) [ 8788.871017] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8788.875883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8788.880008] Lustre: Skipped 1 previous similar message [ 8792.075713] Lustre: server umount lustre-OST0000 complete [ 8793.880503] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8793.893850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8793.896962] Lustre: Skipped 2 previous similar messages [ 8798.884686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8798.888830] Lustre: Skipped 1 previous similar message [ 8799.757756] Lustre: server umount lustre-MDT0000 complete [ 8801.310204] LustreError: 9981:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709062496 with bad export cookie 6066206089486211689 [ 8801.311484] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8801.329066] LustreError: 9981:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8801.519038] Lustre: server umount lustre-MDT0001 complete [ 8806.479846] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing unload_modules_local [ 8807.562735] Key type lgssc unregistered [ 8807.657034] LNet: 22016:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8807.660540] LNet: Removed LNI 192.168.204.153@tcp [ 8822.742543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8822.785461] alg: No test for adler32 (adler32-zlib) [ 8823.656355] Lustre: Lustre: Build Version: 2.15.61_76_g2ee32a4 [ 8823.734587] LNet: Added LNI 192.168.204.153@tcp [8/256/0/180] [ 8823.737447] LNet: Accept secure, port 988 [ 8825.296717] Key type lgssc registered [ 8825.662366] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8829.693151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8832.944164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8835.555139] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 8835.564755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8838.900231] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 8838.912403] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8841.806608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8841.830079] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8841.840968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8842.993373] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8843.008062] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 8843.055370] Lustre: lustre-MDT0000: new disk, initializing [ 8843.103405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8843.110045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8844.498032] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 8850.621123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8850.660926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8850.703875] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8850.729627] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 8850.733452] Lustre: Skipped 1 previous similar message [ 8850.784164] Lustre: lustre-MDT0001: new disk, initializing [ 8850.830057] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8850.855799] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8850.863001] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8852.133646] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 8855.496314] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8856.456058] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8859.347834] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 8859.355071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8859.379372] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 8859.389151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8859.531190] Lustre: lustre-OST0000: new disk, initializing [ 8859.533371] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 8859.570227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8861.819829] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 8865.209288] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8866.728743] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8868.305917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8868.310422] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8868.345454] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8868.909823] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 8870.202330] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8870.303651] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8873.332577] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8873.332909] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8873.335378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8873.354202] LustreError: Skipped 1 previous similar message [ 8877.266088] Lustre: server umount lustre-OST0000 complete [ 8880.854915] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8880.862837] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8880.869497] Lustre: Skipped 1 previous similar message [ 8880.879211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8880.881735] Lustre: Skipped 1 previous similar message [ 8883.348146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8883.348757] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8883.361734] Lustre: Skipped 1 previous similar message [ 8885.098456] Lustre: server umount lustre-MDT0000 complete [ 8887.082843] LustreError: 29352:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709062582 with bad export cookie 16403446143555440716 [ 8887.085283] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8887.103549] LustreError: 29352:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8887.277138] Lustre: server umount lustre-MDT0001 complete [ 8894.309355] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 14:36:30 (1709062590) [ 8897.543607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8897.836239] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8897.903233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8899.895590] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 8902.904130] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8903.975650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8904.186937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8905.648814] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 8907.754105] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8908.805249] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8912.296144] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 8912.303627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8912.564582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8915.113216] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 8917.638123] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8929.634604] LustreError: 1026:0:(llog.c:436:llog_init_handle()) llog type is not specified! [ 8930.050950] Lustre: Modifying parameter general.timeout in log params [ 8932.371869] Lustre: Modifying parameter general.timeout in log params [ 8932.373445] Lustre: Skipped 4 previous similar messages [ 8936.630753] Lustre: Modifying parameter general.timeout in log params [ 8936.632793] Lustre: Skipped 9 previous similar messages [ 8944.690251] Lustre: Modifying parameter general.timeout in log params [ 8944.691878] Lustre: Skipped 18 previous similar messages [ 8960.945132] Lustre: Modifying parameter general.timeout in log params [ 8960.947622] Lustre: Skipped 37 previous similar messages [ 8993.146345] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 14:38:08 (1709062688) [ 8994.631609] Lustre: Setting parameter general.jobid_var in log params [ 8994.633691] Lustre: Skipped 57 previous similar messages [ 8999.332505] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8999.337000] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8999.347425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9004.133958] Lustre: server umount lustre-MDT0000 complete [ 9005.709845] LustreError: 30939:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709062700 with bad export cookie 16403446143555441556 [ 9005.710673] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9005.717251] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9005.717821] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9005.717822] Lustre: Skipped 2 previous similar messages [ 9005.718269] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9005.718270] Lustre: Skipped 3 previous similar messages [ 9005.756422] LustreError: 30939:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9005.886285] Lustre: server umount lustre-MDT0001 complete [ 9013.621931] Lustre: server umount lustre-OST0000 complete [ 9017.096361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9018.906153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9020.699029] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 9020.711335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9022.569592] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 9022.578103] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9025.634146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9025.763236] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9025.881802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9027.202343] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 9031.082753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9031.139022] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9032.505498] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 9034.023119] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9034.913656] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9037.826872] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 9037.834319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9037.922461] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9040.782189] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 9042.837511] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9049.318349] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9049.324742] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9049.333508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9052.463725] Lustre: server umount lustre-OST0000 complete [ 9060.191851] Lustre: server umount lustre-MDT0000 complete [ 9061.983567] LustreError: 16739:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709062757 with bad export cookie 16403446143555443747 [ 9061.984806] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9061.995821] LustreError: 16739:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9062.165796] Lustre: server umount lustre-MDT0001 complete [ 9067.330595] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing unload_modules_local [ 9068.519374] Key type lgssc unregistered [ 9068.629183] LNet: 17538:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9068.632681] LNet: Removed LNI 192.168.204.153@tcp [ 9082.144355] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9082.182134] alg: No test for adler32 (adler32-zlib) [ 9083.081346] Lustre: Lustre: Build Version: 2.15.61_76_g2ee32a4 [ 9083.161473] LNet: Added LNI 192.168.204.153@tcp [8/256/0/180] [ 9083.163992] LNet: Accept secure, port 988 [ 9084.720400] Key type lgssc registered [ 9085.391082] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9089.115301] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9089.130469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9090.337519] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9090.369549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9091.623164] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 9095.809367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9096.109436] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9097.491837] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 9099.186958] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9100.127873] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9103.716506] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 9103.730118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9104.081593] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9106.600082] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 9108.350579] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9121.389774] Lustre: Modifying parameter general.jobid_var in log params [ 9125.092703] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9125.093601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9125.103732] Lustre: Skipped 1 previous similar message [ 9130.102130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9130.104818] Lustre: Skipped 4 previous similar messages [ 9132.268656] Lustre: server umount lustre-MDT0000 complete [ 9133.932388] LustreError: 19423:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709062829 with bad export cookie 17779448729146410698 [ 9133.933866] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9133.943987] LustreError: 19423:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9134.137446] Lustre: server umount lustre-MDT0001 complete [ 9145.848019] Lustre: 23149:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1709062831/real 1709062831] req@ffff88009696ca80 x1792082211519488/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1709062841 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 9145.966936] Lustre: server umount lustre-OST0000 complete [ 9149.893235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9151.546734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9153.200326] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 9153.207238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9154.864974] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 9154.871295] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9157.880946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9158.009105] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9158.036448] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 9158.163692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9159.513455] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 9163.201363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9163.266152] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9163.281585] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 9163.533256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9164.909968] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 9166.687379] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9167.650225] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9171.547379] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 9171.553371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9171.663866] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9171.856194] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9174.114591] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 9175.876842] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9176.857097] Lustre: Modifying parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 9183.214388] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9183.216580] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9183.220762] Lustre: Skipped 1 previous similar message [ 9183.230430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9186.025615] Lustre: server umount lustre-OST0000 complete [ 9188.245720] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9188.251866] Lustre: Skipped 1 previous similar message [ 9188.254605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9188.259206] Lustre: Skipped 2 previous similar messages [ 9193.253620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9193.257947] Lustre: Skipped 1 previous similar message [ 9193.932578] Lustre: server umount lustre-MDT0000 complete [ 9195.589774] LustreError: 25048:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709062890 with bad export cookie 17779448729146411510 [ 9195.594829] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9195.601108] LustreError: 25048:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9195.738008] Lustre: server umount lustre-MDT0001 complete [ 9200.291924] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing unload_modules_local [ 9201.262989] Key type lgssc unregistered [ 9201.362439] LNet: 29004:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9201.369944] LNet: Removed LNI 192.168.204.153@tcp [ 9241.008027] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9241.104665] alg: No test for adler32 (adler32-zlib) [ 9242.095270] Lustre: Lustre: Build Version: 2.15.61_76_g2ee32a4 [ 9242.247204] LNet: Added LNI 192.168.204.153@tcp [8/256/0/180] [ 9242.250574] LNet: Accept secure, port 988 [ 9243.841690] Key type lgssc registered [ 9244.382268] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9244.843802] LustreError: 2665:0:(libcfs_fail.h:221:cfs_race_wait()) cfs_race id 60d sleeping [ 9251.809401] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing load_modules_local [ 9254.815072] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9266.489736] LustreError: 2665:0:(libcfs_fail.h:224:cfs_race_wait()) cfs_fail_race id 60d awake: rc=0 [ 9266.533079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9267.832392] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9267.873003] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9269.130184] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 9275.010988] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 14:42:50 (1709062970) [ 9277.219730] LustreError: 3830:0:(osp_object.c:637:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 9277.223985] LustreError: 3830:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 9277.228155] LustreError: 3830:0:(lod_dev.c:525:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 9277.368659] Lustre: server umount lustre-MDT0000 complete [ 9282.528399] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing unload_modules_local [ 9283.538903] Key type lgssc unregistered [ 9283.667586] LNet: 5910:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9283.675933] LNet: Removed LNI 192.168.204.153@tcp [ 9291.505823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9291.572890] alg: No test for adler32 (adler32-zlib) [ 9292.456731] Lustre: Lustre: Build Version: 2.15.61_76_g2ee32a4 [ 9292.538830] LNet: Added LNI 192.168.204.153@tcp [8/256/0/180] [ 9292.549687] LNet: Accept secure, port 988 [ 9294.093680] Key type lgssc registered [ 9294.491130] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9297.462310] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9297.468456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9298.671593] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9298.706954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9299.809348] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 9302.910359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9303.160910] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9304.226638] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 9305.614397] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9306.447583] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9309.313750] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 9309.324144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9309.556221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9311.501688] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 9313.071765] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9355.012467] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 14:44:10 (1709063050) [ 9355.673293] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 9358.694687] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 14:44:14 (1709063054) [ 9363.141623] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9363.147331] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9363.158609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9365.899245] Lustre: server umount lustre-MDT0000 complete [ 9367.771350] LustreError: 7147:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709063062 with bad export cookie 227126975224507591 [ 9367.773501] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9367.781303] LustreError: 7147:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9367.912508] Lustre: server umount lustre-MDT0001 complete [ 9379.676677] Lustre: 11592:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1709063064/real 1709063064] req@ffff8801307bc000 x1792082430674240/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1709063074 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 9379.866866] Lustre: server umount lustre-OST0000 complete [ 9384.956196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9385.190827] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9385.269015] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9386.546601] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 9390.276245] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9390.413675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9390.773657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9392.332767] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 9394.415999] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9395.572049] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9398.565842] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 9398.573390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9401.536981] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 9401.546076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9401.576958] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 9401.584832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9401.666675] LustreError: 140-5: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 9401.671039] LustreError: 12410:0:(mgs_handler.c:518:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 9401.685633] LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 9401.709442] LustreError: 14603:0:(tgt_mount.c:2223:server_fill_super()) Unable to start targets: -98 [ 9401.712471] LustreError: 14603:0:(tgt_mount.c:1755:server_put_super()) no obd lustre-OST0000 [ 9401.715701] LustreError: 14603:0:(tgt_mount.c:135:server_deregister_mount()) lustre-OST0000 not registered [ 9401.790400] Lustre: server umount lustre-OST0000 complete [ 9401.791935] LustreError: 14603:0:(super25.c:188:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 9405.447110] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 9405.453606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9405.548089] LustreError: 140-5: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 9405.550875] LustreError: 12411:0:(mgs_handler.c:518:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 9405.560461] LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 9405.565103] LustreError: 15215:0:(tgt_mount.c:2223:server_fill_super()) Unable to start targets: -98 [ 9405.569338] LustreError: 15215:0:(tgt_mount.c:1755:server_put_super()) no obd lustre-OST0000 [ 9405.572241] LustreError: 15215:0:(tgt_mount.c:135:server_deregister_mount()) lustre-OST0000 not registered [ 9405.630251] Lustre: server umount lustre-OST0000 complete [ 9405.632146] LustreError: 15215:0:(super25.c:188:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 9406.809496] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 9406.817824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9409.604222] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 9409.611122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9409.684680] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9409.687636] Lustre: Found index 0 for lustre-OST0000, updating log [ 9409.691201] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 9409.713501] Lustre: lustre-OST0000: new disk, initializing [ 9409.716132] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 9409.780235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9411.598822] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 9413.057787] Lustre: Failing over lustre-OST0000 [ 9413.152772] Lustre: server umount lustre-OST0000 complete [ 9415.652596] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9415.656199] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9415.663087] Lustre: Skipped 2 previous similar messages [ 9415.669036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9415.670787] Lustre: Skipped 2 previous similar messages [ 9416.783552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9416.787991] Lustre: Skipped 2 previous similar messages [ 9421.796245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9421.800063] Lustre: Skipped 1 previous similar message [ 9424.818676] Lustre: 16827:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1709063109/real 1709063109] req@ffff8800a7f15500 x1792082430689216/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1709063119 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 9424.960283] Lustre: server umount lustre-MDT0000 complete [ 9426.805466] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9426.816638] LustreError: Skipped 1 previous similar message [ 9426.831378] LustreError: 12403:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709063122 with bad export cookie 227126975224510342 [ 9426.836726] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9427.001208] Lustre: server umount lustre-MDT0001 complete [ 9433.539667] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 14:45:29 (1709063129) [ 9438.403617] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing load_modules_local [ 9441.364463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9441.546383] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9441.587414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9442.724138] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 9446.062449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9447.694000] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 9450.820337] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 9450.829963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9451.039127] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 9452.031730] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 9452.049024] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 9452.053225] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.153@tcp (at 0@lo) [ 9452.059701] Lustre: lustre-OST0000: deleting orphan objects from 0x280000401:163 to 0x280000401:194 [ 9453.198681] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 9456.812429] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 9456.820410] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9456.857505] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 9456.865247] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9456.889372] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9456.955325] Lustre: lustre-OST0001: new disk, initializing [ 9456.958907] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 9458.931152] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 9462.435481] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9463.665081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9463.669847] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9463.716416] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9468.783313] Lustre: Setting parameter general.lod.*.mdt_hash in log params [ 9474.787717] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.53@tcp (stopping) [ 9476.070036] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 9476.081487] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9476.087244] Lustre: Skipped 1 previous similar message [ 9478.726254] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9480.774438] Lustre: server umount lustre-MDT0001 complete [ 9481.829189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9483.733243] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9483.738853] LustreError: Skipped 2 previous similar messages [ 9484.241711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9484.291980] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9484.294931] Lustre: Found index 1 for lustre-MDT0001, updating log [ 9484.318482] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 9484.509697] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9484.512650] Lustre: Skipped 3 previous similar messages [ 9485.671072] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 9486.210966] LustreError: 23967:0:(obd_mount.c:208:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 9486.217551] LustreError: 23967:0:(tgt_mount.c:683:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 9486.222135] LustreError: 19128:0:(mgc_request.c:619:do_requeue()) failed processing log: -17 [ 9486.255338] LustreError: 23972:0:(obd_config.c:2021:class_config_llog_handler()) MGC192.168.204.153@tcp: cfg command failed: rc = -17 [ 9486.258886] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 9489.512148] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 192.168.204.153@tcp (at 0@lo) [ 9492.361922] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 14:46:28 (1709063188) [ 9498.766554] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing load_modules_local [ 9515.590172] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9516.979066] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 9549.433652] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 9549.436647] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9549.447579] Lustre: Skipped 1 previous similar message [ 9549.449394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9549.451402] Lustre: Skipped 5 previous similar messages [ 9555.507600] Lustre: server umount lustre-MDT0000 complete [ 9557.036540] LustreError: 19118:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709063252 with bad export cookie 227126975224511259 [ 9557.037975] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9557.045359] LustreError: 19118:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 9557.178799] Lustre: server umount lustre-MDT0001 complete [ 9568.676138] Lustre: 29080:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1709063253/real 1709063253] req@ffff88009e2d8700 x1792082430784128/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1709063263 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 9568.718217] Lustre: server umount lustre-OST0000 complete [ 9580.593708] Lustre: 29354:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1709063265/real 1709063265] req@ffff8800923b0000 x1792082430784704/t0(0) o39->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1709063275 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 9580.694027] Lustre: server umount lustre-OST0001 complete [ 9582.573441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9585.581702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9585.903716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9588.993696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9592.302894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9592.707857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9598.986589] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing load_modules_local [ 9602.317411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9602.325116] Lustre: lustre-MDT0000: reset Object Index mappings [ 9602.562917] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9602.594861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9603.983986] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 9607.384701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9607.392666] Lustre: lustre-MDT0001: reset Object Index mappings [ 9609.179132] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 9613.174952] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 9613.181913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9615.736453] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 9617.461575] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:200 to 0x280000400:225 [ 9620.098901] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 9620.104930] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9621.723910] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:200 to 0x2c0000400:225 [ 9623.472770] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 9627.737336] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000401:156 to 0x2c0000401:193 [ 9627.737780] Lustre: lustre-OST0000: deleting orphan objects from 0x280000401:349 to 0x280000401:386 [ 9633.605198] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9637.904446] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 9667.721570] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 14:49:23 (1709063363) [ 9682.772453] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9682.776947] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9682.780596] Lustre: Skipped 3 previous similar messages [ 9682.785197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9682.787438] Lustre: Skipped 7 previous similar messages [ 9685.363418] Lustre: server umount lustre-MDT0000 complete [ 9687.390775] LustreError: 5035:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709063382 with bad export cookie 227126975224557781 [ 9687.391966] LustreError: 166-1: MGC192.168.204.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9687.401903] LustreError: 5035:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9687.831668] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9687.835422] LustreError: Skipped 2 previous similar messages [ 9693.661489] Lustre: server umount lustre-MDT0001 complete [ 9701.738764] Lustre: server umount lustre-OST0000 complete [ 9709.856244] Lustre: server umount lustre-OST0001 complete [ 9713.027891] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_hostid [ 9716.603390] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing load_modules_local [ 9720.860648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9723.835702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9726.179845] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 9726.188373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9728.572671] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 9728.586343] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9731.501965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9731.556052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9731.660833] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 9731.677797] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 9731.726966] Lustre: lustre-MDT0000: new disk, initializing [ 9731.806094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9731.809072] Lustre: Skipped 3 previous similar messages [ 9731.818680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9733.417545] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 9737.041276] Lustre: server umount lustre-MDT0000 complete [ 9738.346691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9740.957145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9740.996561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9741.175748] Lustre: Found index 0 for lustre-MDT0000, updating log [ 9741.180646] Lustre: Modifying parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 9742.389045] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 9744.383613] Lustre: server umount lustre-MDT0000 complete [ 9751.492844] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 14:50:47 (1709063447) [ 9752.366847] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 9756.689847] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 14:50:52 (1709063452) [ 9787.965023] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 14:51:23 (1709063483) [ 9794.465212] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_hostid [ 9797.832777] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing load_modules_local [ 9801.774901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9805.403427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9807.946188] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 9807.958492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9810.991587] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 9811.009038] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9814.962697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9814.994460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9815.350607] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 9815.352492] Lustre: Skipped 1 previous similar message [ 9815.385692] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 9815.388336] Lustre: Skipped 1 previous similar message [ 9815.507891] Lustre: lustre-MDT0000: new disk, initializing [ 9815.589805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9816.997128] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 9822.334849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9822.362584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9822.415068] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 9822.417374] Lustre: Skipped 1 previous similar message [ 9822.455289] Lustre: lustre-MDT0001: new disk, initializing [ 9822.501170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9822.505587] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9823.680469] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 9827.096843] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9828.162975] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9831.383952] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 9831.393602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9831.440819] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 9831.455635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9831.611923] Lustre: lustre-OST0000: new disk, initializing [ 9831.613713] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 9832.871263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9832.874737] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9832.917310] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9834.180206] Lustre: DEBUG MARKER: oleg453-server.virtnet: executing set_default_debug -1 all [ 9837.684886] Lustre: DEBUG MARKER: oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9842.554766] Lustre: lustre-MDD0000: changelog on [ 9844.059471] Lustre: lustre-MDD0001: changelog on [ 9861.077189] Lustre: lustre-MDT0000: Client 07d7a489-27b2-4c12-b26b-8462979a4bc9 (at 192.168.204.53@tcp) reconnecting [ 9872.008692] Lustre: DEBUG MARKER: conf-sanity test_135: @@@@@@ FAIL: creation thread is running. Is changelog reader stuck? [ 9873.490405] Lustre: lustre-MDT0000: Client 07d7a489-27b2-4c12-b26b-8462979a4bc9 (at 192.168.204.53@tcp) reconnecting