== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 02:26:22 (1787120782) start mds service on oleg211-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1050: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg211-server' oleg211-server: oleg211-server.virtnet: executing load_modules_local oleg211-server: Loading modules from /home/green/git/lustre-release/lustre oleg211-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1050: echo: write error: Device or resource busy oleg211-server: detected 4 online CPUs by sysfs oleg211-server: MODOPTS_LIBCFS= oleg211-server: Force libcfs to create 2 CPU partitions oleg211-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg211-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg211-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg211-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-OST0000 oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg211-client.virtnet: -o user_xattr,flock 192.168.202.111@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 248 (time 1787120863.62 total 10.01 last 24.78) - open/close 520 (time 1787120874.14 total 20.53 last 25.84) - open/close 753 (time 1787120884.32 total 30.70 last 22.90) - open/close 978 (time 1787120895.06 total 41.45 last 20.94) - open/close 1201 (time 1787120905.96 total 52.35 last 20.47) - open/close 1469 (time 1787120916.44 total 62.82 last 25.58) - open/close 1714 (time 1787120926.44 total 72.83 last 24.49) - open/close 1931 (time 1787120939.12 total 85.51 last 17.11) - open/close 2106 (time 1787120951.42 total 97.81 last 14.23) - open/close 2342 (time 1787120962.11 total 108.49 last 22.08) - open/close 2558 (time 1787120972.11 total 118.50 last 21.59) - open/close 2778 (time 1787120982.25 total 128.63 last 21.70) - open/close 3011 (time 1787120993.05 total 139.44 last 21.56) - open/close 3160 (time 1787121003.25 total 149.64 last 14.61) - open/close 3330 (time 1787121013.42 total 159.81 last 16.72) - open/close 3542 (time 1787121023.71 total 170.10 last 20.60) - open/close 3685 (time 1787121034.82 total 181.21 last 12.88) - open/close 3867 (time 1787121044.84 total 191.23 last 18.16) - open/close 4024 (time 1787121055.40 total 201.79 last 14.87) - open/close 4200 (time 1787121065.50 total 211.89 last 17.42) - open/close 4441 (time 1787121076.33 total 222.72 last 22.27) - open/close 4668 (time 1787121086.95 total 233.34 last 21.37) - open/close 4885 (time 1787121097.84 total 244.22 last 19.94) - open/close 5059 (time 1787121108.19 total 254.57 last 16.81) - open/close 5182 (time 1787121118.89 total 265.28 last 11.49) - open/close 5440 (time 1787121128.94 total 275.33 last 25.66) - open/close 5693 (time 1787121139.75 total 286.14 last 23.42) - open/close 5912 (time 1787121150.31 total 296.70 last 20.73) - open/close 6140 (time 1787121160.48 total 306.87 last 22.42) - open/close 6403 (time 1787121170.51 total 316.90 last 26.23) - open/close 6592 (time 1787121180.91 total 327.30 last 18.16) - open/close 6808 (time 1787121190.92 total 337.30 last 21.59) - open/close 6931 (time 1787121201.31 total 347.70 last 11.83) - open/close 7164 (time 1787121211.31 total 357.70 last 23.30) - open/close 7350 (time 1787121221.80 total 368.19 last 17.74) - open/close 7465 (time 1787121231.83 total 378.22 last 11.47) - open/close 7674 (time 1787121243.10 total 389.49 last 18.54) - open/close 7874 (time 1787121253.12 total 399.51 last 19.96) - open/close 8072 (time 1787121263.12 total 409.51 last 19.79) - open/close 8294 (time 1787121273.14 total 419.53 last 22.17) - open/close 8641 (time 1787121283.52 total 429.91 last 33.43) - open/close 8914 (time 1787121294.51 total 440.90 last 24.84) - open/close 9138 (time 1787121304.51 total 450.90 last 22.39) - open/close 9306 (time 1787121315.04 total 461.42 last 15.96) - open/close 9552 (time 1787121325.05 total 471.43 last 24.58) - open/close 9799 (time 1787121335.44 total 481.83 last 23.76) - open/close 10000 (time 1787121343.22 total 489.61 last 25.85) - open/close 10221 (time 1787121353.62 total 500.01 last 21.24) - open/close 10456 (time 1787121363.94 total 510.33 last 22.77) - open/close 10613 (time 1787121373.96 total 520.35 last 15.68) - open/close 10848 (time 1787121384.43 total 530.82 last 22.43) - open/close 11094 (time 1787121395.31 total 541.70 last 22.62) - open/close 11363 (time 1787121406.34 total 552.73 last 24.38) - open/close 11561 (time 1787121416.34 total 562.73 last 19.79) - open/close 11811 (time 1787121427.10 total 573.49 last 23.24) - open/close 11997 (time 1787121437.13 total 583.51 last 18.55) - open/close 12102 (time 1787121449.00 total 595.39 last 8.84) - open/close 12222 (time 1787121459.49 total 605.88 last 11.44) - open/close 12313 (time 1787121469.50 total 615.89 last 9.09) - open/close 12461 (time 1787121479.92 total 626.31 last 14.21) - open/close 12636 (time 1787121490.27 total 636.66 last 16.91) - open/close 12843 (time 1787121500.71 total 647.09 last 19.83) - open/close 12886 (time 1787121511.20 total 657.59 last 4.10) - open/close 13023 (time 1787121521.21 total 667.59 last 13.69) - open/close 13142 (time 1787121531.39 total 677.78 last 11.68) - open/close 13286 (time 1787121541.88 total 688.27 last 13.73) - open/close 13394 (time 1787121552.30 total 698.69 last 10.37) - open/close 13552 (time 1787121563.25 total 709.64 last 14.43) - open/close 13808 (time 1787121573.25 total 719.64 last 25.59) - open/close 14051 (time 1787121584.03 total 730.41 last 22.56) - open/close 14320 (time 1787121594.04 total 740.43 last 26.86) - open/close 14598 (time 1787121604.04 total 750.43 last 27.79) - open/close 14808 (time 1787121614.07 total 760.45 last 20.95) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-14865) error: No space left on device total: 14865 open/close in 761.42 seconds: 19.52 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 9017: kill: (76057) - No such process - unlinked 0 (time 1787121620 ; total 0 ; last 0) - unlinked 10000 (time 1787121720 ; total 100 ; last 100) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-14865) error: No such file or directory total: 14865 unlinks in 141 seconds: 105.425529 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg211-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg211-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg211-server stop mds service on oleg211-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg211-server unloading modules via unload_modules_local on: 'oleg211-server' oleg211-server: oleg211-server.virtnet: executing unload_modules_local oleg211-server: modules unloaded. pdsh@oleg211-client: oleg211-client: ssh exited with exit code 2 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2 pdsh@oleg211-client: oleg211-client: ssh exited with exit code 2