== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 16:23:18 (1717014198) start mds service on oleg311-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg311-server' oleg311-server: oleg311-server.virtnet: executing load_modules_local oleg311-server: Loading modules from /home/green/git/lustre-release/lustre oleg311-server: detected 4 online CPUs by sysfs oleg311-server: Force libcfs to create 2 CPU partitions oleg311-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg311-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg311-server: oleg311-server.virtnet: executing set_default_debug -1 all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg311-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg311-server: oleg311-server.virtnet: executing set_default_debug -1 all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg311-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg311-server: oleg311-server.virtnet: executing set_default_debug -1 all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Started lustre-OST0000 oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg311-client.virtnet: -o user_xattr,flock oleg311-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 335 (time 1717014255.47 total 10.05 last 33.32) - open/close 621 (time 1717014265.48 total 20.06 last 28.58) - open/close 798 (time 1717014276.38 total 30.97 last 16.23) - open/close 1011 (time 1717014286.39 total 40.97 last 21.29) - open/close 1335 (time 1717014296.40 total 50.98 last 32.38) - open/close 1587 (time 1717014306.43 total 61.02 last 25.11) - open/close 1886 (time 1717014316.96 total 71.55 last 28.39) - open/close 2181 (time 1717014326.97 total 81.56 last 29.47) - open/close 2505 (time 1717014337.16 total 91.75 last 31.80) - open/close 2861 (time 1717014347.41 total 102.00 last 34.72) - open/close 3040 (time 1717014357.43 total 112.01 last 17.88) - open/close 3233 (time 1717014367.44 total 122.02 last 19.27) - open/close 3446 (time 1717014377.46 total 132.05 last 21.25) - open/close 3727 (time 1717014387.48 total 142.06 last 28.07) - open/close 3894 (time 1717014398.34 total 152.93 last 15.37) - open/close 3991 (time 1717014408.55 total 163.14 last 9.50) - open/close 4224 (time 1717014419.30 total 173.89 last 21.67) - open/close 4403 (time 1717014432.23 total 186.82 last 13.85) - open/close 4585 (time 1717014443.24 total 197.83 last 16.53) - open/close 4886 (time 1717014453.83 total 208.42 last 28.43) - open/close 5212 (time 1717014463.94 total 218.53 last 32.25) - open/close 5531 (time 1717014474.67 total 229.25 last 29.74) - open/close 5743 (time 1717014485.10 total 239.68 last 20.33) - open/close 6095 (time 1717014495.76 total 250.34 last 33.02) - open/close 6359 (time 1717014506.68 total 261.27 last 24.17) - open/close 6532 (time 1717014516.69 total 271.27 last 17.29) - open/close 6779 (time 1717014529.33 total 283.91 last 19.54) - open/close 6977 (time 1717014539.42 total 294.01 last 19.61) - open/close 7283 (time 1717014550.28 total 304.87 last 28.17) - open/close 7539 (time 1717014560.92 total 315.51 last 24.06) - open/close 7812 (time 1717014572.35 total 326.94 last 23.89) - open/close 7998 (time 1717014583.08 total 337.67 last 17.33) - open/close 8271 (time 1717014593.71 total 348.29 last 25.70) - open/close 8549 (time 1717014604.42 total 359.00 last 25.95) - open/close 8770 (time 1717014614.68 total 369.26 last 21.54) - open/close 9065 (time 1717014625.45 total 380.04 last 27.38) - open/close 9368 (time 1717014635.97 total 390.55 last 28.82) - open/close 9553 (time 1717014646.41 total 401.00 last 17.72) - open/close 9923 (time 1717014656.42 total 411.00 last 36.98) - open/close 10000 (time 1717014659.04 total 413.63 last 29.29) - open/close 10283 (time 1717014669.05 total 423.64 last 28.28) - open/close 10579 (time 1717014680.06 total 434.65 last 26.88) - open/close 10878 (time 1717014690.80 total 445.39 last 27.84) - open/close 11188 (time 1717014701.80 total 456.39 last 28.17) - open/close 11474 (time 1717014712.66 total 467.25 last 26.34) - open/close 12456 (time 1717014722.67 total 477.25 last 98.17) - open/close 13934 (time 1717014732.67 total 487.25 last 147.77) - open/close 15491 (time 1717014742.67 total 497.26 last 155.63) - open/close 17025 (time 1717014752.67 total 507.26 last 153.39) - open/close 18508 (time 1717014762.67 total 517.26 last 148.29) - open/close 20000 (time 1717014772.38 total 526.96 last 153.79) - open/close 21578 (time 1717014782.38 total 536.96 last 157.75) - open/close 23080 (time 1717014792.38 total 546.97 last 150.11) - open/close 24501 (time 1717014802.39 total 556.97 last 142.09) - open/close 25836 (time 1717014812.40 total 566.98 last 133.35) - open/close 27359 (time 1717014822.40 total 576.98 last 152.28) - open/close 28748 (time 1717014832.40 total 586.98 last 138.90) - open/close 30000 (time 1717014840.98 total 595.57 last 145.85) - open/close 31436 (time 1717014850.99 total 605.57 last 143.56) - open/close 33058 (time 1717014860.99 total 615.58 last 162.10) - open/close 34536 (time 1717014870.99 total 625.58 last 147.76) - open/close 36020 (time 1717014881.00 total 635.58 last 148.36) - open/close 37621 (time 1717014891.00 total 645.58 last 160.07) - open/close 39069 (time 1717014901.00 total 655.59 last 144.76) - open/close 40000 (time 1717014907.75 total 662.34 last 137.93) - open/close 41448 (time 1717014917.75 total 672.34 last 144.74) - open/close 42968 (time 1717014927.76 total 682.34 last 151.99) - open/close 44420 (time 1717014937.76 total 692.34 last 145.19) - open/close 45756 (time 1717014947.77 total 702.35 last 133.46) - open/close 47069 (time 1717014957.77 total 712.35 last 131.27) - open/close 48492 (time 1717014967.77 total 722.36 last 142.23) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 730.37 seconds: 67.95 ops/second - unlinked 0 (time 1717014977 ; total 0 ; last 0) - unlinked 10000 (time 1717015013 ; total 36 ; last 36) - unlinked 20000 (time 1717015054 ; total 77 ; last 41) - unlinked 30000 (time 1717015099 ; total 122 ; last 45) - unlinked 40000 (time 1717015135 ; total 158 ; last 36) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 195 seconds: 254.523071 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg311-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg311-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg311-server stop mds service on oleg311-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg311-server stop mds service on oleg311-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg311-server unloading modules on: 'oleg311-server' oleg311-server: oleg311-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg311-client: oleg311-client: ssh exited with exit code 2 pdsh@oleg311-client: oleg311-server: ssh exited with exit code 2 pdsh@oleg311-client: oleg311-client: ssh exited with exit code 2 pdsh@oleg311-client: oleg311-server: ssh exited with exit code 2