== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 00:39:02 (1734673142) start mds service on oleg132-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg132-server' oleg132-server: oleg132-server.virtnet: executing load_modules_local oleg132-server: Loading modules from /home/green/git/lustre-release/lustre oleg132-server: detected 4 online CPUs by sysfs oleg132-server: MODOPTS_LIBCFS= oleg132-server: Force libcfs to create 2 CPU partitions oleg132-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg132-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg132-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg132-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-OST0000 oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg132-client.virtnet: -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 302 (time 1734673200.07 total 10.39 last 29.06) - open/close 558 (time 1734673210.13 total 20.45 last 25.45) - open/close 792 (time 1734673220.14 total 30.46 last 23.38) - open/close 1085 (time 1734673230.81 total 41.13 last 27.46) - open/close 1296 (time 1734673241.68 total 52.00 last 19.41) - open/close 1486 (time 1734673252.69 total 63.01 last 17.25) - open/close 1596 (time 1734673262.91 total 73.23 last 10.77) - open/close 1857 (time 1734673273.60 total 83.92 last 24.41) - open/close 2086 (time 1734673283.99 total 94.31 last 22.03) - open/close 2277 (time 1734673294.08 total 104.40 last 18.93) - open/close 2386 (time 1734673304.45 total 114.77 last 10.51) - open/close 2511 (time 1734673315.13 total 125.45 last 11.70) - open/close 2677 (time 1734673325.54 total 135.86 last 15.95) - open/close 2780 (time 1734673336.13 total 146.46 last 9.73) - open/close 3000 (time 1734673346.96 total 157.28 last 20.32) - open/close 3135 (time 1734673357.87 total 168.19 last 12.37) - open/close 3308 (time 1734673368.80 total 179.13 last 15.82) - open/close 3596 (time 1734673379.58 total 189.90 last 26.73) - open/close 3831 (time 1734673389.78 total 200.10 last 23.03) - open/close 4008 (time 1734673399.79 total 210.11 last 17.69) - open/close 4225 (time 1734673410.06 total 220.38 last 21.13) - open/close 4475 (time 1734673420.28 total 230.60 last 24.46) - open/close 4764 (time 1734673430.88 total 241.20 last 27.27) - open/close 5046 (time 1734673441.44 total 251.76 last 26.69) - open/close 5246 (time 1734673452.19 total 262.51 last 18.61) - open/close 5581 (time 1734673463.15 total 273.47 last 30.56) - open/close 5898 (time 1734673474.00 total 284.32 last 29.24) - open/close 6183 (time 1734673484.36 total 294.69 last 27.49) - open/close 6446 (time 1734673494.71 total 305.03 last 25.43) - open/close 6663 (time 1734673504.79 total 315.11 last 21.53) - open/close 6872 (time 1734673514.80 total 325.12 last 20.88) - open/close 7026 (time 1734673525.37 total 335.69 last 14.57) - open/close 7316 (time 1734673535.95 total 346.27 last 27.39) - open/close 7585 (time 1734673546.32 total 356.65 last 25.93) - open/close 7742 (time 1734673556.60 total 366.92 last 15.29) - open/close 7946 (time 1734673567.37 total 377.70 last 18.92) - open/close 8147 (time 1734673578.28 total 388.60 last 18.44) - open/close 8397 (time 1734673588.42 total 398.75 last 24.63) - open/close 8688 (time 1734673599.13 total 409.45 last 27.18) - open/close 8912 (time 1734673609.20 total 419.52 last 22.26) - open/close 9172 (time 1734673620.05 total 430.37 last 23.97) - open/close 9366 (time 1734673630.99 total 441.31 last 17.73) - open/close 9628 (time 1734673641.15 total 451.47 last 25.79) - open/close 9789 (time 1734673651.71 total 462.03 last 15.24) - open/close 10000 (time 1734673656.07 total 466.39 last 48.43) - open/close 11453 (time 1734673666.07 total 476.40 last 145.20) - open/close 12737 (time 1734673676.08 total 486.41 last 128.27) - open/close 13967 (time 1734673686.09 total 496.41 last 122.94) - open/close 15386 (time 1734673696.09 total 506.42 last 141.83) - open/close 16898 (time 1734673706.10 total 516.42 last 151.16) - open/close 18293 (time 1734673716.10 total 526.42 last 139.49) - open/close 20000 (time 1734673725.61 total 535.93 last 179.45) - open/close 21342 (time 1734673735.61 total 545.93 last 134.18) - open/close 22671 (time 1734673745.61 total 555.93 last 132.90) - open/close 24028 (time 1734673755.62 total 565.94 last 135.59) - open/close 25365 (time 1734673765.62 total 575.95 last 133.66) - open/close 26915 (time 1734673775.63 total 585.95 last 154.91) - open/close 28447 (time 1734673785.63 total 595.95 last 153.17) - open/close 29873 (time 1734673795.64 total 605.96 last 142.52) - open/close 31655 (time 1734673805.64 total 615.96 last 178.20) - open/close 32961 (time 1734673815.64 total 625.97 last 130.52) - open/close 34300 (time 1734673825.64 total 635.97 last 133.89) - open/close 35738 (time 1734673835.65 total 645.97 last 143.78) - open/close 37064 (time 1734673845.65 total 655.97 last 132.52) - open/close 38731 (time 1734673855.65 total 665.98 last 166.67) - open/close 40000 (time 1734673862.06 total 672.38 last 198.24) - open/close 41881 (time 1734673872.06 total 682.38 last 188.08) - open/close 44262 (time 1734673882.06 total 692.39 last 237.93) - open/close 45907 (time 1734673892.07 total 702.39 last 164.47) - open/close 47415 (time 1734673902.07 total 712.39 last 150.78) - open/close 49413 (time 1734673912.07 total 722.39 last 199.75) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 723.75 seconds: 68.58 ops/second - unlinked 0 (time 1734673914 ; total 0 ; last 0) - unlinked 10000 (time 1734673958 ; total 44 ; last 44) - unlinked 20000 (time 1734673989 ; total 75 ; last 31) - unlinked 30000 (time 1734674018 ; total 104 ; last 29) - unlinked 40000 (time 1734674052 ; total 138 ; last 34) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 178 seconds: 278.831451 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg132-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg132-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg132-server stop mds service on oleg132-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg132-server stop mds service on oleg132-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg132-server unloading modules via unload_modules_local on: 'oleg132-server' oleg132-server: oleg132-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg132-client: oleg132-client: ssh exited with exit code 2 pdsh@oleg132-client: oleg132-server: ssh exited with exit code 2 pdsh@oleg132-client: oleg132-client: ssh exited with exit code 2 pdsh@oleg132-client: oleg132-server: ssh exited with exit code 2