== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 02:49:28 (1772437768) start mds service on oleg242-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing load_modules_local oleg242-server: Loading modules from /home/green/git/lustre-release/lustre oleg242-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg242-server: detected 4 online CPUs by sysfs oleg242-server: MODOPTS_LIBCFS= oleg242-server: Force libcfs to create 2 CPU partitions oleg242-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg242-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg242-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg242-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started lustre-OST0000 oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg242-client.virtnet: -o user_xattr,flock 192.168.202.142@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 315 (time 1772437812.60 total 10.66 last 29.54) - open/close 577 (time 1772437823.03 total 21.10 last 25.11) - open/close 743 (time 1772437833.34 total 31.40 last 16.11) - open/close 935 (time 1772437843.91 total 41.98 last 18.16) - open/close 1308 (time 1772437856.58 total 54.65 last 29.44) - open/close 1689 (time 1772437867.45 total 65.51 last 35.06) - open/close 1873 (time 1772437877.75 total 75.82 last 17.86) - open/close 2392 (time 1772437888.12 total 86.19 last 50.04) - open/close 3067 (time 1772437898.87 total 96.93 last 62.81) - open/close 3525 (time 1772437909.81 total 107.88 last 41.83) - open/close 3968 (time 1772437920.75 total 118.82 last 40.50) - open/close 4297 (time 1772437931.51 total 129.57 last 30.60) - open/close 4917 (time 1772437942.01 total 140.08 last 59.03) - open/close 5407 (time 1772437952.01 total 150.08 last 49.00) - open/close 5610 (time 1772437962.61 total 160.68 last 19.14) - open/close 5881 (time 1772437973.23 total 171.30 last 25.52) - open/close 6415 (time 1772437983.48 total 181.54 last 52.14) - open/close 6879 (time 1772437995.25 total 193.32 last 39.40) - open/close 7472 (time 1772438005.26 total 203.32 last 59.28) - open/close 8025 (time 1772438015.47 total 213.54 last 54.12) - open/close 8629 (time 1772438025.47 total 223.54 last 60.40) - open/close 9118 (time 1772438036.34 total 234.41 last 45.01) - open/close 9518 (time 1772438047.03 total 245.09 last 37.42) - open/close 10000 (time 1772438055.68 total 253.75 last 55.71) - open/close 10545 (time 1772438065.68 total 263.75 last 54.48) - open/close 11095 (time 1772438075.90 total 273.96 last 53.84) - open/close 11567 (time 1772438085.90 total 283.96 last 47.20) - open/close 12122 (time 1772438096.63 total 294.69 last 51.73) - open/close 12616 (time 1772438107.57 total 305.64 last 45.12) - open/close 12921 (time 1772438118.39 total 316.46 last 28.20) - open/close 13414 (time 1772438130.49 total 328.56 last 40.74) - open/close 13903 (time 1772438140.53 total 338.60 last 48.71) - open/close 16614 (time 1772438150.53 total 348.60 last 271.03) - open/close 18715 (time 1772438160.54 total 358.61 last 209.92) - open/close 20000 (time 1772438166.60 total 364.66 last 212.16) - open/close 22901 (time 1772438176.60 total 374.67 last 290.07) - open/close 26352 (time 1772438186.60 total 384.67 last 345.01) - open/close 29845 (time 1772438196.60 total 394.67 last 349.17) - open/close 33565 (time 1772438206.61 total 404.67 last 371.92) - open/close 37573 (time 1772438216.61 total 414.68 last 400.72) - open/close 40000 (time 1772438222.99 total 421.06 last 380.23) - open/close 44143 (time 1772438232.99 total 431.06 last 414.24) - open/close 47725 (time 1772438242.99 total 441.06 last 358.14) total: 50000 open/close in 446.74 seconds: 111.92 ops/second - unlinked 0 (time 1772438250 ; total 0 ; last 0) - unlinked 10000 (time 1772438264 ; total 14 ; last 14) - unlinked 20000 (time 1772438280 ; total 30 ; last 16) - unlinked 30000 (time 1772438292 ; total 42 ; last 12) - unlinked 40000 (time 1772438306 ; total 56 ; last 14) total: 50000 unlinks in 70 seconds: 714.285706 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg242-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg242-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg242-server stop mds service on oleg242-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg242-server unloading modules via unload_modules_local on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing unload_modules_local oleg242-server: modules unloaded. pdsh@oleg242-client: oleg242-client: ssh exited with exit code 2 pdsh@oleg242-client: oleg242-server: ssh exited with exit code 2 pdsh@oleg242-client: oleg242-client: ssh exited with exit code 2 pdsh@oleg242-client: oleg242-server: ssh exited with exit code 2