== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 02:59:02 (1727765942) start mds service on oleg412-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg412-server' oleg412-server: oleg412-server.virtnet: executing load_modules_local oleg412-server: Loading modules from /home/green/git/lustre-release/lustre oleg412-server: detected 4 online CPUs by sysfs oleg412-server: Force libcfs to create 2 CPU partitions oleg412-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg412-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg412-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg412-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1 Started lustre-OST0000 oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg412-client.virtnet: -o user_xattr,flock 192.168.204.112@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 250 (time 1727766002.06 total 10.91 last 22.91) - open/close 597 (time 1727766012.07 total 20.91 last 34.69) - open/close 801 (time 1727766022.13 total 30.98 last 20.27) - open/close 1004 (time 1727766032.56 total 41.41 last 19.47) - open/close 1271 (time 1727766043.04 total 51.89 last 25.47) - open/close 1544 (time 1727766054.03 total 62.87 last 24.85) - open/close 1765 (time 1727766064.27 total 73.12 last 21.58) - open/close 1844 (time 1727766074.27 total 83.12 last 7.90) - open/close 1962 (time 1727766084.48 total 93.32 last 11.57) - open/close 2120 (time 1727766095.43 total 104.28 last 14.42) - open/close 2276 (time 1727766105.43 total 114.28 last 15.60) - open/close 2458 (time 1727766115.95 total 124.80 last 17.30) - open/close 2600 (time 1727766126.47 total 135.32 last 13.50) - open/close 2816 (time 1727766136.86 total 145.71 last 20.79) - open/close 3039 (time 1727766149.35 total 158.19 last 17.86) - open/close 3142 (time 1727766159.35 total 168.20 last 10.30) - open/close 3283 (time 1727766169.94 total 178.79 last 13.32) - open/close 3382 (time 1727766180.28 total 189.12 last 9.58) - open/close 3461 (time 1727766191.24 total 200.08 last 7.21) - open/close 3707 (time 1727766201.74 total 210.59 last 23.41) - open/close 3906 (time 1727766211.86 total 220.71 last 19.67) - open/close 4109 (time 1727766221.87 total 230.72 last 20.29) - open/close 4305 (time 1727766232.56 total 241.40 last 18.34) - open/close 4570 (time 1727766243.05 total 251.90 last 25.25) - open/close 4902 (time 1727766253.06 total 261.90 last 33.18) - open/close 5163 (time 1727766263.69 total 272.53 last 24.55) - open/close 5436 (time 1727766273.79 total 282.63 last 27.03) - open/close 5697 (time 1727766285.11 total 293.96 last 23.05) - open/close 5893 (time 1727766295.71 total 304.55 last 18.50) - open/close 6080 (time 1727766306.12 total 314.96 last 17.96) - open/close 6289 (time 1727766316.22 total 325.07 last 20.68) - open/close 6553 (time 1727766326.74 total 335.59 last 25.09) - open/close 6834 (time 1727766337.14 total 345.99 last 27.02) - open/close 7107 (time 1727766347.15 total 356.00 last 27.28) - open/close 7457 (time 1727766357.15 total 366.00 last 34.99) - open/close 7690 (time 1727766368.09 total 376.94 last 21.30) - open/close 7886 (time 1727766379.53 total 388.38 last 17.13) - open/close 8139 (time 1727766390.52 total 399.37 last 23.02) - open/close 8384 (time 1727766400.66 total 409.50 last 24.18) - open/close 8737 (time 1727766411.63 total 420.47 last 32.18) - open/close 8940 (time 1727766422.31 total 431.16 last 19.00) - open/close 9161 (time 1727766432.82 total 441.67 last 21.02) - open/close 9303 (time 1727766443.80 total 452.65 last 12.94) - open/close 9483 (time 1727766454.09 total 462.94 last 17.50) - open/close 10000 (time 1727766460.04 total 468.88 last 86.90) - open/close 12200 (time 1727766470.04 total 478.89 last 219.97) - open/close 14823 (time 1727766480.04 total 488.89 last 262.25) - open/close 17653 (time 1727766490.04 total 498.89 last 282.94) - open/close 20000 (time 1727766499.19 total 508.04 last 256.62) - open/close 22534 (time 1727766509.19 total 518.04 last 253.34) - open/close 24940 (time 1727766519.20 total 528.04 last 240.48) - open/close 26987 (time 1727766529.20 total 538.05 last 204.66) - open/close 28945 (time 1727766539.22 total 548.07 last 195.41) - open/close 30000 (time 1727766544.03 total 552.88 last 219.05) - open/close 32184 (time 1727766554.04 total 562.89 last 218.25) - open/close 34621 (time 1727766564.04 total 572.89 last 243.69) - open/close 36450 (time 1727766574.05 total 582.89 last 182.84) - open/close 38107 (time 1727766584.05 total 592.90 last 165.64) - open/close 39911 (time 1727766594.05 total 602.90 last 180.37) - open/close 41846 (time 1727766604.05 total 612.90 last 193.47) - open/close 43568 (time 1727766614.06 total 622.90 last 172.14) - open/close 45326 (time 1727766624.06 total 632.91 last 175.73) - open/close 47016 (time 1727766634.06 total 642.91 last 168.94) - open/close 48791 (time 1727766644.07 total 652.91 last 177.47) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 657.91 seconds: 75.44 ops/second - unlinked 0 (time 1727766650 ; total 0 ; last 0) - unlinked 10000 (time 1727766680 ; total 30 ; last 30) - unlinked 20000 (time 1727766709 ; total 59 ; last 29) - unlinked 30000 (time 1727766741 ; total 91 ; last 32) - unlinked 40000 (time 1727766771 ; total 121 ; last 30) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 148 seconds: 335.351349 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg412-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg412-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg412-server stop mds service on oleg412-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg412-server stop mds service on oleg412-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg412-server unloading modules on: 'oleg412-server' oleg412-server: oleg412-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg412-client: oleg412-client: ssh exited with exit code 2 pdsh@oleg412-client: oleg412-server: ssh exited with exit code 2 pdsh@oleg412-client: oleg412-client: ssh exited with exit code 2 pdsh@oleg412-client: oleg412-server: ssh exited with exit code 2