== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:49:12 (1786978152) start mds service on oleg115-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg115-server' oleg115-server: oleg115-server.virtnet: executing load_modules_local oleg115-server: Loading modules from /home/green/git/lustre-release/lustre oleg115-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg115-server: detected 4 online CPUs by sysfs oleg115-server: MODOPTS_LIBCFS= oleg115-server: Force libcfs to create 2 CPU partitions oleg115-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg115-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg115-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg115-server: oleg115-server.virtnet: executing set_default_debug -1 all pdsh@oleg115-client: oleg115-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg115-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg115-server: oleg115-server.virtnet: executing set_default_debug -1 all pdsh@oleg115-client: oleg115-server: ssh exited with exit code 1 Started lustre-OST0000 oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg115-client.virtnet: -o user_xattr,flock 192.168.201.115@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 193 (time 1786978243.57 total 10.33 last 18.69) - open/close 407 (time 1786978253.72 total 20.48 last 21.09) - open/close 655 (time 1786978263.75 total 30.50 last 24.74) - open/close 830 (time 1786978274.67 total 41.42 last 16.02) - open/close 1108 (time 1786978285.72 total 52.48 last 25.15) - open/close 1246 (time 1786978295.73 total 62.49 last 13.79) - open/close 1464 (time 1786978305.75 total 72.50 last 21.76) - open/close 1718 (time 1786978315.76 total 82.51 last 25.37) - open/close 1952 (time 1786978325.77 total 92.53 last 23.37) - open/close 2190 (time 1786978336.70 total 103.45 last 21.78) - open/close 2391 (time 1786978346.86 total 113.61 last 19.78) - open/close 2573 (time 1786978356.87 total 123.62 last 18.19) - open/close 2841 (time 1786978367.69 total 134.44 last 24.77) - open/close 3089 (time 1786978377.71 total 144.47 last 24.74) - open/close 3333 (time 1786978388.52 total 155.28 last 22.57) - open/close 3595 (time 1786978398.69 total 165.45 last 25.76) - open/close 3880 (time 1786978409.26 total 176.02 last 26.97) - open/close 4097 (time 1786978419.26 total 186.02 last 21.69) - open/close 4212 (time 1786978429.48 total 196.23 last 11.26) - open/close 4441 (time 1786978439.48 total 206.24 last 22.89) - open/close 4626 (time 1786978450.10 total 216.85 last 17.43) - open/close 4749 (time 1786978460.13 total 226.88 last 12.26) - open/close 5010 (time 1786978470.13 total 236.89 last 26.09) - open/close 5203 (time 1786978480.24 total 246.99 last 19.09) - open/close 5456 (time 1786978490.24 total 257.00 last 25.28) - open/close 5651 (time 1786978500.33 total 267.08 last 19.34) - open/close 5877 (time 1786978510.35 total 277.10 last 22.56) - open/close 6116 (time 1786978520.36 total 287.11 last 23.88) - open/close 6393 (time 1786978530.59 total 297.35 last 27.06) - open/close 6614 (time 1786978540.85 total 307.61 last 21.54) - open/close 6754 (time 1786978551.64 total 318.40 last 12.98) - open/close 6870 (time 1786978561.69 total 328.45 last 11.54) - open/close 6998 (time 1786978572.59 total 339.35 last 11.75) - open/close 7184 (time 1786978582.76 total 349.52 last 18.29) - open/close 7372 (time 1786978593.72 total 360.47 last 17.16) - open/close 7559 (time 1786978603.73 total 370.48 last 18.68) - open/close 7821 (time 1786978614.75 total 381.50 last 23.78) - open/close 8067 (time 1786978624.99 total 391.75 last 24.02) - open/close 8218 (time 1786978635.49 total 402.25 last 14.38) - open/close 8468 (time 1786978645.56 total 412.31 last 24.84) - open/close 8658 (time 1786978655.94 total 422.69 last 18.31) - open/close 8867 (time 1786978666.55 total 433.31 last 19.69) - open/close 9048 (time 1786978676.58 total 443.33 last 18.05) - open/close 9237 (time 1786978686.58 total 453.34 last 18.89) - open/close 9318 (time 1786978698.67 total 465.43 last 6.70) - open/close 9597 (time 1786978708.69 total 475.44 last 27.85) - open/close 9773 (time 1786978719.13 total 485.89 last 16.85) - open/close 10000 (time 1786978727.33 total 494.08 last 27.71) - open/close 10192 (time 1786978738.04 total 504.79 last 17.93) - open/close 10349 (time 1786978748.05 total 514.81 last 15.67) - open/close 10440 (time 1786978759.08 total 525.83 last 8.25) - open/close 10572 (time 1786978769.70 total 536.46 last 12.42) - open/close 10762 (time 1786978779.72 total 546.47 last 18.97) - open/close 10856 (time 1786978789.73 total 556.49 last 9.39) - open/close 11044 (time 1786978799.75 total 566.50 last 18.78) - open/close 11286 (time 1786978810.38 total 577.13 last 22.76) - open/close 11458 (time 1786978820.39 total 587.15 last 17.18) - open/close 11694 (time 1786978830.41 total 597.17 last 23.55) - open/close 11892 (time 1786978841.51 total 608.26 last 17.84) - open/close 12124 (time 1786978851.51 total 618.27 last 23.20) - open/close 12328 (time 1786978861.51 total 628.27 last 20.40) - open/close 12524 (time 1786978871.53 total 638.28 last 19.57) - open/close 12727 (time 1786978881.54 total 648.29 last 20.28) - open/close 12911 (time 1786978891.54 total 658.30 last 18.39) - open/close 13124 (time 1786978901.54 total 668.30 last 21.30) - open/close 13319 (time 1786978912.24 total 678.99 last 18.23) - open/close 13421 (time 1786978922.24 total 689.00 last 10.20) - open/close 13490 (time 1786978932.32 total 699.08 last 6.84) - open/close 13640 (time 1786978942.33 total 709.09 last 14.98) - open/close 13853 (time 1786978952.95 total 719.70 last 20.07) - open/close 14060 (time 1786978963.38 total 730.13 last 19.84) - open/close 14211 (time 1786978973.80 total 740.56 last 14.48) - open/close 14308 (time 1786978984.74 total 751.50 last 8.87) - open/close 14364 (time 1786978994.75 total 761.50 last 5.60) - open/close 14517 (time 1786979005.32 total 772.08 last 14.47) - open/close 14633 (time 1786979016.22 total 782.98 last 10.64) - open/close 14763 (time 1786979026.48 total 793.23 last 12.67) - open/close 14816 (time 1786979037.17 total 803.93 last 4.96) - open/close 15024 (time 1786979047.40 total 814.16 last 20.34) - open/close 15200 (time 1786979057.41 total 824.16 last 17.59) - open/close 15257 (time 1786979068.26 total 835.01 last 5.25) - open/close 15433 (time 1786979078.46 total 845.21 last 17.26) - open/close 15605 (time 1786979088.46 total 855.21 last 17.20) - open/close 15749 (time 1786979099.24 total 865.99 last 13.36) - open/close 15913 (time 1786979109.25 total 876.00 last 16.38) - open/close 16097 (time 1786979121.12 total 887.87 last 15.50) - open/close 16324 (time 1786979131.14 total 897.90 last 22.65) - open/close 16479 (time 1786979141.15 total 907.91 last 15.48) - open/close 16625 (time 1786979151.18 total 917.94 last 14.56) - open/close 16724 (time 1786979161.65 total 928.41 last 9.45) - open/close 16862 (time 1786979171.66 total 938.42 last 13.79) - open/close 17025 (time 1786979181.67 total 948.42 last 16.30) - open/close 17190 (time 1786979191.67 total 958.42 last 16.50) - open/close 17301 (time 1786979202.61 total 969.36 last 10.15) - open/close 17452 (time 1786979215.67 total 982.43 last 11.56) - open/close 17589 (time 1786979225.84 total 992.59 last 13.48) - open/close 17725 (time 1786979235.85 total 1002.61 last 13.58) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-17809) error: No space left on device total: 17809 open/close in 1009.62 seconds: 17.64 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 9017: kill: (76161) - No such process - unlinked 0 (time 1786979249 ; total 0 ; last 0) - unlinked 10000 (time 1786979405 ; total 156 ; last 156) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-17809) error: No such file or directory total: 17809 unlinks in 254 seconds: 70.114174 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg115-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg115-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg115-server stop mds service on oleg115-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg115-server unloading modules via unload_modules_local on: 'oleg115-server' oleg115-server: oleg115-server.virtnet: executing unload_modules_local oleg115-server: modules unloaded. pdsh@oleg115-client: oleg115-client: ssh exited with exit code 2 pdsh@oleg115-client: oleg115-server: ssh exited with exit code 2 pdsh@oleg115-client: oleg115-client: ssh exited with exit code 2 pdsh@oleg115-client: oleg115-server: ssh exited with exit code 2