== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 21:12:13 (1767319933) start mds service on oleg242-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing load_modules_local oleg242-server: Loading modules from /home/green/git/lustre-release/lustre oleg242-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg242-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg242-server: detected 4 online CPUs by sysfs oleg242-server: MODOPTS_LIBCFS= oleg242-server: Force libcfs to create 2 CPU partitions oleg242-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg242-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg242-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg242-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started lustre-OST0000 oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg242-client.virtnet: -o user_xattr,flock 192.168.202.142@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 242 (time 1767320090.77 total 10.52 last 23.01) - open/close 473 (time 1767320100.78 total 20.53 last 23.07) - open/close 553 (time 1767320110.81 total 30.56 last 7.97) - open/close 731 (time 1767320120.82 total 40.56 last 17.79) - open/close 897 (time 1767320131.85 total 51.60 last 15.04) - open/close 1123 (time 1767320141.86 total 61.60 last 22.59) - open/close 1379 (time 1767320151.88 total 71.63 last 25.53) - open/close 1624 (time 1767320161.89 total 81.64 last 24.49) - open/close 1825 (time 1767320172.62 total 92.37 last 18.74) - open/close 2059 (time 1767320182.74 total 102.49 last 23.11) - open/close 2341 (time 1767320192.82 total 112.56 last 28.00) - open/close 2569 (time 1767320202.82 total 122.57 last 22.79) - open/close 2749 (time 1767320212.82 total 132.57 last 18.00) - open/close 2948 (time 1767320223.11 total 142.86 last 19.33) - open/close 3130 (time 1767320233.29 total 153.04 last 17.87) - open/close 3251 (time 1767320245.15 total 164.90 last 10.20) - open/close 3502 (time 1767320255.19 total 174.94 last 25.01) - open/close 3734 (time 1767320265.86 total 185.61 last 21.74) - open/close 3999 (time 1767320275.99 total 195.73 last 26.17) - open/close 4259 (time 1767320286.42 total 206.17 last 24.91) - open/close 4468 (time 1767320296.47 total 216.22 last 20.80) - open/close 4697 (time 1767320306.96 total 226.71 last 21.83) - open/close 4878 (time 1767320316.97 total 236.72 last 18.08) - open/close 5150 (time 1767320326.98 total 246.73 last 27.18) - open/close 5372 (time 1767320336.99 total 256.74 last 22.18) - open/close 5609 (time 1767320347.94 total 267.69 last 21.65) - open/close 5914 (time 1767320358.87 total 278.62 last 27.90) - open/close 6165 (time 1767320369.10 total 288.85 last 24.52) - open/close 6438 (time 1767320379.11 total 298.86 last 27.28) - open/close 6818 (time 1767320389.11 total 308.86 last 37.99) - open/close 7087 (time 1767320399.83 total 319.58 last 25.11) - open/close 7255 (time 1767320411.53 total 331.28 last 14.36) - open/close 7457 (time 1767320422.39 total 342.14 last 18.60) - open/close 7684 (time 1767320432.41 total 352.16 last 22.65) - open/close 7957 (time 1767320443.22 total 362.96 last 25.27) - open/close 8222 (time 1767320453.77 total 373.52 last 25.11) - open/close 8524 (time 1767320463.77 total 383.52 last 30.19) - open/close 8764 (time 1767320473.78 total 393.53 last 23.97) - open/close 9006 (time 1767320484.69 total 404.44 last 22.18) - open/close 9234 (time 1767320494.70 total 414.45 last 22.79) - open/close 9454 (time 1767320504.72 total 424.47 last 21.95) - open/close 9710 (time 1767320515.72 total 435.47 last 23.27) - open/close 9871 (time 1767320526.41 total 446.16 last 15.06) - open/close 9999 (time 1767320536.41 total 456.16 last 12.80) - open/close 10128 (time 1767320546.43 total 466.18 last 12.88) - open/close 10345 (time 1767320556.44 total 476.19 last 21.67) - open/close 10476 (time 1767320567.44 total 487.19 last 11.91) - open/close 10704 (time 1767320577.68 total 497.43 last 22.25) - open/close 11004 (time 1767320587.69 total 507.43 last 29.99) - open/close 11227 (time 1767320597.71 total 517.46 last 22.25) - open/close 11508 (time 1767320607.73 total 527.48 last 28.04) - open/close 11726 (time 1767320618.21 total 537.95 last 20.81) - open/close 11838 (time 1767320629.13 total 548.88 last 10.25) - open/close 12043 (time 1767320639.77 total 559.52 last 19.27) - open/close 12342 (time 1767320649.79 total 569.54 last 29.84) - open/close 12551 (time 1767320659.80 total 579.55 last 20.87) - open/close 12889 (time 1767320669.81 total 589.56 last 33.79) - open/close 13201 (time 1767320679.82 total 599.57 last 31.17) - open/close 13433 (time 1767320690.07 total 609.82 last 22.62) - open/close 13710 (time 1767320700.10 total 619.85 last 27.63) - open/close 13957 (time 1767320710.85 total 630.60 last 22.97) - open/close 14167 (time 1767320722.67 total 642.42 last 17.77) - open/close 14378 (time 1767320733.66 total 653.41 last 19.19) - open/close 14555 (time 1767320744.17 total 663.92 last 16.84) - open/close 14781 (time 1767320754.25 total 674.00 last 22.42) - open/close 14978 (time 1767320764.26 total 684.01 last 19.68) - open/close 15209 (time 1767320774.67 total 694.41 last 22.20) - open/close 15474 (time 1767320785.50 total 705.25 last 24.46) - open/close 15723 (time 1767320795.52 total 715.27 last 24.84) - open/close 15940 (time 1767320805.79 total 725.53 last 21.14) - open/close 16114 (time 1767320816.22 total 735.97 last 16.67) - open/close 16275 (time 1767320826.59 total 746.34 last 15.52) - open/close 16511 (time 1767320837.47 total 757.22 last 21.70) - open/close 16759 (time 1767320848.08 total 767.83 last 23.37) - open/close 17016 (time 1767320858.21 total 777.96 last 25.37) - open/close 17111 (time 1767320868.37 total 788.12 last 9.36) - open/close 17307 (time 1767320878.85 total 798.60 last 18.69) - open/close 17560 (time 1767320888.88 total 808.63 last 25.24) - open/close 17774 (time 1767320899.75 total 819.50 last 19.69) - open/close 18060 (time 1767320910.62 total 830.37 last 26.31) - open/close 18336 (time 1767320920.63 total 840.37 last 27.58) - open/close 18520 (time 1767320931.29 total 851.04 last 17.25) - open/close 18760 (time 1767320941.94 total 861.68 last 22.55) - open/close 19060 (time 1767320952.02 total 871.76 last 29.76) - open/close 19276 (time 1767320962.06 total 881.80 last 21.51) - open/close 19428 (time 1767320972.70 total 892.45 last 14.28) - open/close 19815 (time 1767320982.70 total 902.45 last 38.70) - open/close 20000 (time 1767320984.92 total 904.67 last 83.38) - open/close 20804 (time 1767320994.94 total 914.69 last 80.21) - open/close 21521 (time 1767321004.95 total 924.70 last 71.65) - open/close 22345 (time 1767321014.95 total 934.70 last 82.39) - open/close 23131 (time 1767321024.96 total 944.71 last 78.55) - open/close 23902 (time 1767321034.96 total 954.71 last 77.08) - open/close 24722 (time 1767321044.97 total 964.72 last 81.95) - open/close 25541 (time 1767321054.98 total 974.73 last 81.78) - open/close 26306 (time 1767321064.99 total 984.73 last 76.47) - open/close 27131 (time 1767321075.01 total 994.75 last 82.34) - open/close 27870 (time 1767321085.01 total 1004.76 last 73.87) - open/close 28610 (time 1767321095.01 total 1014.76 last 73.98) - open/close 29356 (time 1767321105.03 total 1024.78 last 74.47) - open/close 30000 (time 1767321113.71 total 1033.46 last 74.22) - open/close 30776 (time 1767321123.71 total 1043.46 last 77.55) - open/close 31590 (time 1767321133.71 total 1053.46 last 81.39) - open/close 32424 (time 1767321143.71 total 1063.46 last 83.40) - open/close 33274 (time 1767321153.72 total 1073.46 last 84.99) - open/close 33998 (time 1767321163.72 total 1083.46 last 72.40) - open/close 34844 (time 1767321173.72 total 1093.46 last 84.60) - open/close 35617 (time 1767321183.72 total 1103.47 last 77.28) - open/close 36390 (time 1767321193.73 total 1113.48 last 77.24) - open/close 37157 (time 1767321203.73 total 1123.48 last 76.66) - open/close 37927 (time 1767321213.76 total 1133.51 last 76.77) - open/close 38635 (time 1767321223.77 total 1143.52 last 70.76) - open/close 39398 (time 1767321233.77 total 1153.52 last 76.29) - open/close 40000 (time 1767321241.41 total 1161.16 last 78.80) - open/close 40763 (time 1767321251.41 total 1171.16 last 76.29) - open/close 41626 (time 1767321261.41 total 1181.16 last 86.27) - open/close 42376 (time 1767321271.42 total 1191.17 last 74.96) - open/close 43139 (time 1767321281.42 total 1201.17 last 76.29) - open/close 43860 (time 1767321291.42 total 1211.17 last 72.09) - open/close 44600 (time 1767321301.44 total 1221.19 last 73.85) - open/close 45352 (time 1767321311.44 total 1231.19 last 75.18) - open/close 46057 (time 1767321321.46 total 1241.21 last 70.41) - open/close 46818 (time 1767321331.46 total 1251.21 last 76.07) - open/close 47555 (time 1767321341.47 total 1261.22 last 73.63) - open/close 48283 (time 1767321351.48 total 1271.23 last 72.70) - open/close 48964 (time 1767321361.49 total 1281.24 last 68.06) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 1291.17 seconds: 38.44 ops/second - unlinked 0 (time 1767321376 ; total 0 ; last 0) - unlinked 10000 (time 1767321507 ; total 131 ; last 131) - unlinked 20000 (time 1767321634 ; total 258 ; last 127) - unlinked 30000 (time 1767321762 ; total 386 ; last 128) - unlinked 40000 (time 1767321888 ; total 512 ; last 126) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 646 seconds: 76.829720 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg242-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg242-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg242-server stop mds service on oleg242-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg242-server stop mds service on oleg242-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg242-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing unload_modules_local oleg242-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg242-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg242-client: oleg242-client: ssh exited with exit code 2 pdsh@oleg242-client: oleg242-server: ssh exited with exit code 2 pdsh@oleg242-client: oleg242-client: ssh exited with exit code 2 pdsh@oleg242-client: oleg242-server: ssh exited with exit code 2