== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 15:45:22 (1746128722) start mds service on oleg249-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg249-server' oleg249-server: oleg249-server.virtnet: executing load_modules_local oleg249-server: Loading modules from /home/green/git/lustre-release/lustre oleg249-server: detected 4 online CPUs by sysfs oleg249-server: MODOPTS_LIBCFS= oleg249-server: Force libcfs to create 2 CPU partitions oleg249-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg249-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg249-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg249-server: oleg249-server.virtnet: executing set_default_debug -1 all pdsh@oleg249-client: oleg249-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg249-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg249-server: oleg249-server.virtnet: executing set_default_debug -1 all pdsh@oleg249-client: oleg249-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg249-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg249-server: oleg249-server.virtnet: executing set_default_debug -1 all pdsh@oleg249-client: oleg249-server: ssh exited with exit code 1 Started lustre-OST0000 oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg249-client.virtnet: -o user_xattr,flock 192.168.202.149@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 168 (time 1746128796.30 total 12.23 last 13.74) - open/close 323 (time 1746128807.10 total 23.04 last 14.34) - open/close 407 (time 1746128817.35 total 33.28 last 8.20) - open/close 514 (time 1746128827.98 total 43.91 last 10.06) - open/close 587 (time 1746128838.92 total 54.85 last 6.68) - open/close 790 (time 1746128849.38 total 65.31 last 19.40) - open/close 929 (time 1746128862.32 total 78.26 last 10.74) - open/close 1106 (time 1746128872.70 total 88.64 last 17.05) - open/close 1221 (time 1746128883.37 total 99.30 last 10.78) - open/close 1372 (time 1746128893.48 total 109.41 last 14.94) - open/close 1514 (time 1746128903.48 total 119.41 last 14.19) - open/close 1721 (time 1746128914.23 total 130.17 last 19.25) - open/close 1883 (time 1746128924.56 total 140.49 last 15.69) - open/close 2059 (time 1746128935.85 total 151.79 last 15.59) - open/close 2173 (time 1746128946.53 total 162.46 last 10.68) - open/close 2300 (time 1746128957.18 total 173.11 last 11.93) - open/close 2454 (time 1746128967.21 total 183.15 last 15.34) - open/close 2672 (time 1746128978.16 total 194.09 last 19.92) - open/close 2845 (time 1746128988.40 total 204.34 last 16.88) - open/close 2968 (time 1746128998.97 total 214.90 last 11.64) - open/close 3146 (time 1746129009.95 total 225.88 last 16.21) - open/close 3390 (time 1746129020.69 total 236.62 last 22.71) - open/close 3491 (time 1746129031.14 total 247.07 last 9.67) - open/close 3582 (time 1746129041.38 total 257.31 last 8.89) - open/close 3676 (time 1746129051.64 total 267.58 last 9.16) - open/close 3827 (time 1746129061.66 total 277.59 last 15.08) - open/close 3896 (time 1746129071.73 total 287.66 last 6.85) - open/close 3981 (time 1746129081.78 total 297.71 last 8.46) - open/close 4058 (time 1746129091.78 total 307.71 last 7.70) - open/close 4181 (time 1746129102.34 total 318.27 last 11.64) - open/close 4347 (time 1746129112.81 total 328.75 last 15.85) - open/close 4420 (time 1746129123.78 total 339.72 last 6.65) - open/close 4535 (time 1746129134.37 total 350.30 last 10.87) - open/close 4672 (time 1746129145.87 total 361.81 last 11.90) - open/close 5022 (time 1746129156.03 total 371.96 last 34.47) - open/close 5342 (time 1746129166.03 total 381.97 last 31.98) - open/close 5558 (time 1746129176.04 total 391.97 last 21.60) - open/close 5694 (time 1746129186.67 total 402.61 last 12.78) - open/close 5861 (time 1746129197.46 total 413.39 last 15.48) - open/close 6066 (time 1746129208.18 total 424.11 last 19.13) - open/close 6319 (time 1746129219.14 total 435.07 last 23.08) - open/close 6590 (time 1746129229.68 total 445.61 last 25.71) - open/close 6910 (time 1746129240.37 total 456.30 last 29.94) - open/close 7122 (time 1746129250.46 total 466.40 last 21.00) - open/close 7417 (time 1746129261.31 total 477.25 last 27.19) - open/close 7655 (time 1746129272.19 total 488.13 last 21.87) - open/close 7945 (time 1746129282.91 total 498.85 last 27.05) - open/close 8194 (time 1746129293.23 total 509.16 last 24.14) - open/close 8422 (time 1746129303.28 total 519.21 last 22.68) - open/close 9046 (time 1746129313.28 total 529.21 last 62.39) - open/close 10000 (time 1746129319.97 total 535.91 last 142.58) - open/close 11632 (time 1746129329.98 total 545.91 last 163.05) - open/close 13266 (time 1746129339.98 total 555.92 last 163.35) - open/close 14965 (time 1746129349.99 total 565.92 last 169.86) - open/close 16453 (time 1746129359.99 total 575.93 last 148.72) - open/close 17929 (time 1746129369.99 total 585.93 last 147.56) - open/close 19174 (time 1746129380.00 total 595.94 last 124.40) - open/close 20000 (time 1746129385.17 total 601.10 last 159.81) - open/close 21508 (time 1746129395.17 total 611.10 last 150.80) - open/close 22994 (time 1746129405.17 total 621.11 last 148.55) - open/close 24488 (time 1746129415.18 total 631.11 last 149.37) - open/close 25895 (time 1746129425.19 total 641.12 last 140.57) - open/close 27063 (time 1746129435.19 total 651.12 last 116.77) - open/close 28173 (time 1746129445.20 total 661.13 last 110.89) - open/close 29392 (time 1746129455.20 total 671.14 last 121.82) - open/close 30000 (time 1746129459.56 total 675.49 last 139.60) - open/close 31134 (time 1746129469.56 total 685.49 last 113.38) - open/close 32354 (time 1746129479.56 total 695.50 last 121.97) - open/close 33748 (time 1746129489.57 total 705.50 last 139.35) - open/close 35409 (time 1746129499.57 total 715.50 last 166.06) - open/close 36551 (time 1746129509.57 total 725.50 last 114.20) - open/close 37666 (time 1746129519.57 total 735.51 last 111.46) - open/close 38872 (time 1746129529.59 total 745.52 last 120.40) - open/close 40000 (time 1746129538.52 total 754.45 last 126.37) - open/close 41208 (time 1746129548.52 total 764.45 last 120.78) - open/close 42300 (time 1746129558.52 total 774.46 last 109.14) - open/close 43372 (time 1746129568.54 total 784.47 last 107.07) - open/close 44442 (time 1746129578.54 total 794.47 last 106.95) - open/close 45536 (time 1746129588.55 total 804.48 last 109.26) - open/close 46601 (time 1746129598.56 total 814.50 last 106.39) - open/close 47913 (time 1746129608.57 total 824.50 last 131.16) - open/close 49036 (time 1746129618.57 total 834.50 last 112.26) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 838.97 seconds: 59.16 ops/second - unlinked 0 (time 1746129624 ; total 0 ; last 0) - unlinked 10000 (time 1746129670 ; total 46 ; last 46) - unlinked 20000 (time 1746129706 ; total 82 ; last 36) - unlinked 30000 (time 1746129739 ; total 115 ; last 33) - unlinked 40000 (time 1746129778 ; total 154 ; last 39) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 194 seconds: 255.835052 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg249-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg249-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg249-server stop mds service on oleg249-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg249-server stop mds service on oleg249-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg249-server unloading modules via unload_modules_local on: 'oleg249-server' oleg249-server: oleg249-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg249-client: oleg249-client: ssh exited with exit code 2 pdsh@oleg249-client: oleg249-server: ssh exited with exit code 2 pdsh@oleg249-client: oleg249-client: ssh exited with exit code 2 pdsh@oleg249-client: oleg249-server: ssh exited with exit code 2