== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 17:27:03 (1779312423) start mds service on oleg437-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg437-server' oleg437-server: oleg437-server.virtnet: executing load_modules_local oleg437-server: Loading modules from /home/green/git/lustre-release/lustre oleg437-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg437-server: detected 4 online CPUs by sysfs oleg437-server: MODOPTS_LIBCFS= oleg437-server: Force libcfs to create 2 CPU partitions oleg437-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg437-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg437-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg437-server: oleg437-server.virtnet: executing set_default_debug -1 all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg437-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg437-server: oleg437-server.virtnet: executing set_default_debug -1 all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Started lustre-OST0000 oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg437-client.virtnet: -o user_xattr,flock 192.168.204.137@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 208 (time 1779312491.42 total 10.01 last 20.78) - open/close 370 (time 1779312501.42 total 20.01 last 16.20) - open/close 595 (time 1779312512.08 total 30.67 last 21.10) - open/close 854 (time 1779312522.85 total 41.44 last 24.06) - open/close 1037 (time 1779312532.85 total 51.44 last 18.29) - open/close 1140 (time 1779312542.93 total 61.52 last 10.22) - open/close 1253 (time 1779312552.94 total 71.53 last 11.29) - open/close 1449 (time 1779312563.22 total 81.82 last 19.05) - open/close 1661 (time 1779312573.78 total 92.37 last 20.08) - open/close 1889 (time 1779312583.83 total 102.42 last 22.68) - open/close 2147 (time 1779312594.61 total 113.20 last 23.94) - open/close 2345 (time 1779312605.20 total 123.79 last 18.70) - open/close 2505 (time 1779312615.20 total 133.79 last 15.99) - open/close 2691 (time 1779312625.55 total 144.14 last 17.98) - open/close 2976 (time 1779312635.55 total 154.15 last 28.49) - open/close 3154 (time 1779312646.62 total 165.21 last 16.08) - open/close 3352 (time 1779312656.67 total 175.26 last 19.71) - open/close 3561 (time 1779312667.47 total 186.06 last 19.35) - open/close 3811 (time 1779312678.29 total 196.89 last 23.09) - open/close 4049 (time 1779312688.30 total 206.89 last 23.80) - open/close 4285 (time 1779312698.31 total 216.91 last 23.56) - open/close 4463 (time 1779312708.37 total 226.96 last 17.71) - open/close 4705 (time 1779312719.04 total 237.64 last 22.66) - open/close 4961 (time 1779312729.77 total 248.36 last 23.87) - open/close 5189 (time 1779312740.18 total 258.77 last 21.91) - open/close 5440 (time 1779312750.82 total 269.41 last 23.58) - open/close 5630 (time 1779312760.85 total 279.44 last 18.94) - open/close 5825 (time 1779312772.25 total 290.84 last 17.11) - open/close 6070 (time 1779312782.25 total 300.85 last 24.49) - open/close 6326 (time 1779312793.18 total 311.78 last 23.42) - open/close 6559 (time 1779312803.99 total 322.59 last 21.55) - open/close 6743 (time 1779312815.90 total 334.49 last 15.46) - open/close 6889 (time 1779312826.03 total 344.62 last 14.41) - open/close 7097 (time 1779312836.25 total 354.84 last 20.35) - open/close 7249 (time 1779312849.18 total 367.78 last 11.75) - open/close 7434 (time 1779312859.19 total 377.78 last 18.49) - open/close 7622 (time 1779312869.20 total 387.79 last 18.79) - open/close 7801 (time 1779312879.20 total 397.79 last 17.89) - open/close 7933 (time 1779312891.22 total 409.82 last 10.98) - open/close 8086 (time 1779312901.23 total 419.82 last 15.29) - open/close 8174 (time 1779312911.56 total 430.16 last 8.51) - open/close 8360 (time 1779312921.57 total 440.17 last 18.58) - open/close 8527 (time 1779312931.67 total 450.26 last 16.55) - open/close 8528 (time 1779312941.90 total 460.49 last 0.10) - open/close 8623 (time 1779312951.91 total 470.50 last 9.49) - open/close 8737 (time 1779312961.95 total 480.54 last 11.36) - open/close 8815 (time 1779312971.95 total 490.54 last 7.80) - open/close 8966 (time 1779312983.96 total 502.55 last 12.57) - open/close 9176 (time 1779312994.38 total 512.98 last 20.14) - open/close 9328 (time 1779313004.90 total 523.49 last 14.46) - open/close 9441 (time 1779313015.15 total 533.74 last 11.02) - open/close 9578 (time 1779313025.81 total 544.41 last 12.84) - open/close 9726 (time 1779313035.93 total 554.53 last 14.62) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-9905) error: No space left on device total: 9905 open/close in 562.03 seconds: 17.62 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 8986: kill: (74218) - No such process - unlinked 0 (time 1779313048 ; total 1 ; last 1) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-9905) error: No such file or directory total: 9905 unlinks in 83 seconds: 119.337349 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg437-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg437-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg437-server stop mds service on oleg437-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg437-server unloading modules via unload_modules_local on: 'oleg437-server' oleg437-server: oleg437-server.virtnet: executing unload_modules_local oleg437-server: modules unloaded. pdsh@oleg437-client: oleg437-client: ssh exited with exit code 2 pdsh@oleg437-client: oleg437-server: ssh exited with exit code 2 pdsh@oleg437-client: oleg437-client: ssh exited with exit code 2 pdsh@oleg437-client: oleg437-server: ssh exited with exit code 2