== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 12:47:13 (1772473633) start mds service on oleg246-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg246-server' oleg246-server: oleg246-server.virtnet: executing load_modules_local oleg246-server: Loading modules from /home/green/git/lustre-release/lustre oleg246-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg246-server: detected 4 online CPUs by sysfs oleg246-server: MODOPTS_LIBCFS= oleg246-server: Force libcfs to create 2 CPU partitions oleg246-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg246-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg246-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg246-server: oleg246-server.virtnet: executing set_default_debug -1 all pdsh@oleg246-client: oleg246-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg246-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg246-server: oleg246-server.virtnet: executing set_default_debug -1 all pdsh@oleg246-client: oleg246-server: ssh exited with exit code 1 Started lustre-OST0000 oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg246-client.virtnet: -o user_xattr,flock 192.168.202.146@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 220 (time 1772473706.40 total 10.36 last 21.23) - open/close 468 (time 1772473717.33 total 21.29 last 22.70) - open/close 664 (time 1772473728.09 total 32.05 last 18.21) - open/close 899 (time 1772473738.10 total 42.06 last 23.48) - open/close 1119 (time 1772473748.17 total 52.13 last 21.84) - open/close 1324 (time 1772473758.92 total 62.88 last 19.08) - open/close 1545 (time 1772473769.61 total 73.57 last 20.68) - open/close 1781 (time 1772473779.62 total 83.58 last 23.58) - open/close 1974 (time 1772473790.02 total 93.98 last 18.56) - open/close 2185 (time 1772473800.20 total 104.16 last 20.73) - open/close 2430 (time 1772473810.52 total 114.48 last 23.73) - open/close 2685 (time 1772473821.35 total 125.31 last 23.56) - open/close 2901 (time 1772473831.36 total 135.32 last 21.58) - open/close 3083 (time 1772473841.36 total 145.32 last 18.19) - open/close 3352 (time 1772473851.38 total 155.34 last 26.85) - open/close 3534 (time 1772473861.39 total 165.35 last 18.18) - open/close 3720 (time 1772473871.41 total 175.37 last 18.56) - open/close 3904 (time 1772473881.41 total 185.37 last 18.39) - open/close 4097 (time 1772473892.44 total 196.40 last 17.50) - open/close 4337 (time 1772473902.85 total 206.81 last 23.06) - open/close 4480 (time 1772473913.23 total 217.19 last 13.78) - open/close 4671 (time 1772473923.84 total 227.80 last 17.99) - open/close 4836 (time 1772473934.34 total 238.30 last 15.71) - open/close 5014 (time 1772473945.16 total 249.12 last 16.46) - open/close 5210 (time 1772473956.10 total 260.06 last 17.91) - open/close 5363 (time 1772473966.41 total 270.37 last 14.84) - open/close 5540 (time 1772473977.10 total 281.07 last 16.55) - open/close 5683 (time 1772473987.27 total 291.23 last 14.07) - open/close 5882 (time 1772473997.27 total 301.24 last 19.89) - open/close 6089 (time 1772474007.62 total 311.59 last 20.00) - open/close 6234 (time 1772474018.33 total 322.29 last 13.54) - open/close 6378 (time 1772474028.49 total 332.45 last 14.17) - open/close 6584 (time 1772474039.12 total 343.08 last 19.39) - open/close 6773 (time 1772474049.86 total 353.82 last 17.60) - open/close 6906 (time 1772474059.86 total 363.82 last 13.30) - open/close 7061 (time 1772474070.35 total 374.32 last 14.77) - open/close 7242 (time 1772474080.84 total 384.80 last 17.27) - open/close 7389 (time 1772474090.84 total 394.80 last 14.69) - open/close 7519 (time 1772474100.87 total 404.83 last 12.96) - open/close 7604 (time 1772474111.51 total 415.47 last 7.99) - open/close 7785 (time 1772474122.38 total 426.34 last 16.65) - open/close 7979 (time 1772474132.39 total 436.35 last 19.38) - open/close 8062 (time 1772474142.85 total 446.82 last 7.93) - open/close 8185 (time 1772474152.86 total 456.83 last 12.29) - open/close 8367 (time 1772474163.40 total 467.36 last 17.28) - open/close 8557 (time 1772474174.17 total 478.13 last 17.64) - open/close 8764 (time 1772474184.32 total 488.28 last 20.38) - open/close 8914 (time 1772474194.82 total 498.78 last 14.29) - open/close 9109 (time 1772474205.64 total 509.60 last 18.03) - open/close 9320 (time 1772474216.19 total 520.15 last 19.99) - open/close 9454 (time 1772474226.43 total 530.39 last 13.09) - open/close 9563 (time 1772474237.12 total 541.08 last 10.20) - open/close 9797 (time 1772474247.76 total 551.73 last 21.99) - open/close 9969 (time 1772474258.44 total 562.40 last 16.12) - open/close 10147 (time 1772474269.13 total 573.10 last 16.64) - open/close 10294 (time 1772474279.14 total 583.10 last 14.69) - open/close 10368 (time 1772474289.54 total 593.50 last 7.11) - open/close 10578 (time 1772474299.58 total 603.54 last 20.91) - open/close 10689 (time 1772474310.43 total 614.39 last 10.23) - open/close 10918 (time 1772474321.23 total 625.19 last 21.20) - open/close 11114 (time 1772474331.53 total 635.50 last 19.03) - open/close 11315 (time 1772474341.96 total 645.92 last 19.28) - open/close 11502 (time 1772474351.97 total 655.93 last 18.68) - open/close 11689 (time 1772474362.94 total 666.90 last 17.05) - open/close 11973 (time 1772474372.95 total 676.91 last 28.38) - open/close 12936 (time 1772474382.95 total 686.91 last 96.26) - open/close 13902 (time 1772474392.95 total 696.92 last 96.58) - open/close 14821 (time 1772474402.96 total 706.92 last 91.85) - open/close 15754 (time 1772474412.96 total 716.92 last 93.28) - open/close 16626 (time 1772474422.97 total 726.93 last 87.16) - open/close 17633 (time 1772474432.97 total 736.93 last 100.65) - open/close 18559 (time 1772474442.98 total 746.94 last 92.56) - open/close 19589 (time 1772474452.98 total 756.94 last 102.97) - open/close 20000 (time 1772474457.57 total 761.53 last 89.53) - open/close 20901 (time 1772474467.57 total 771.53 last 90.10) - open/close 21912 (time 1772474477.57 total 781.54 last 101.06) - open/close 22756 (time 1772474487.58 total 791.54 last 84.35) - open/close 23718 (time 1772474497.59 total 801.55 last 96.13) - open/close 24563 (time 1772474507.60 total 811.56 last 84.38) - open/close 25474 (time 1772474517.61 total 821.57 last 91.00) - open/close 26376 (time 1772474527.62 total 831.58 last 90.12) - open/close 27389 (time 1772474537.63 total 841.59 last 101.25) - open/close 28351 (time 1772474547.64 total 851.61 last 96.04) - open/close 29450 (time 1772474557.65 total 861.61 last 109.85) - open/close 30000 (time 1772474563.17 total 867.13 last 99.59) - open/close 30891 (time 1772474573.17 total 877.14 last 89.09) - open/close 31752 (time 1772474583.18 total 887.14 last 86.07) - open/close 32546 (time 1772474593.18 total 897.14 last 79.36) - open/close 33481 (time 1772474603.20 total 907.16 last 93.34) - open/close 34272 (time 1772474613.21 total 917.17 last 79.00) - open/close 35160 (time 1772474623.22 total 927.18 last 88.72) - open/close 36067 (time 1772474633.22 total 937.18 last 90.68) - open/close 37003 (time 1772474643.23 total 947.19 last 93.54) - open/close 37948 (time 1772474653.23 total 957.19 last 94.48) - open/close 38882 (time 1772474663.24 total 967.20 last 93.35) - open/close 39864 (time 1772474673.24 total 977.20 last 98.15) - open/close 40808 (time 1772474683.26 total 987.22 last 94.25) - open/close 41809 (time 1772474693.26 total 997.22 last 100.05) - open/close 42715 (time 1772474703.27 total 1007.23 last 90.52) - open/close 43600 (time 1772474713.28 total 1017.24 last 88.42) - open/close 44536 (time 1772474723.28 total 1027.24 last 93.57) - open/close 45470 (time 1772474733.28 total 1037.24 last 93.39) - open/close 46334 (time 1772474743.29 total 1047.25 last 86.37) - open/close 47251 (time 1772474753.30 total 1057.27 last 91.53) - open/close 48325 (time 1772474763.31 total 1067.27 last 107.35) - open/close 49395 (time 1772474773.32 total 1077.28 last 106.91) total: 50000 open/close in 1083.24 seconds: 46.16 ops/second - unlinked 0 (time 1772474782 ; total 0 ; last 0) - unlinked 10000 (time 1772474877 ; total 95 ; last 95) - unlinked 20000 (time 1772474932 ; total 150 ; last 55) - unlinked 30000 (time 1772474987 ; total 205 ; last 55) - unlinked 40000 (time 1772475038 ; total 256 ; last 51) total: 50000 unlinks in 306 seconds: 163.398697 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg246-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg246-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg246-server stop mds service on oleg246-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg246-server unloading modules via unload_modules_local on: 'oleg246-server' oleg246-server: oleg246-server.virtnet: executing unload_modules_local oleg246-server: modules unloaded. pdsh@oleg246-client: oleg246-client: ssh exited with exit code 2 pdsh@oleg246-client: oleg246-server: ssh exited with exit code 2 pdsh@oleg246-client: oleg246-client: ssh exited with exit code 2 pdsh@oleg246-client: oleg246-server: ssh exited with exit code 2