-----============= acceptance-small: conf-sanity ============----- Wed May 29 17:27:21 EDT 2024 excepting tests: 32newtarball 110 Stopping clients: oleg128-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg128-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg128-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg128-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg128-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg128-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg128-server oleg128-server: oleg128-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: detected 4 online CPUs by sysfs oleg128-server: Force libcfs to create 2 CPU partitions oleg128-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg128-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg128-server: oleg128-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg128-client: oleg128-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg128-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg128-server: oleg128-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg128-client: oleg128-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg128-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg128-server: oleg128-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg128-client: oleg128-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg128-server: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg128-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg128-server: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg128-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg128-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg128-server stop mds service on oleg128-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg128-server stop mds service on oleg128-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg128-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 17:29:05 (1717018145) Stopping clients: oleg128-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg128-client.virtnet /mnt/lustre2 (opts:-f) oleg128-server: oleg128-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: detected 4 online CPUs by sysfs oleg128-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg128-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg128-server: oleg128-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg128-client: oleg128-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg128-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg128-server: oleg128-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg128-client: oleg128-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg128-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg128-server: oleg128-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg128-client: oleg128-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg128-client.virtnet: -o user_xattr,flock oleg128-server@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 1541 (time 1717018212.00 total 10.00 last 154.07) - open/close 2945 (time 1717018222.00 total 20.01 last 140.34) - open/close 4820 (time 1717018232.02 total 30.02 last 187.24) - open/close 6251 (time 1717018242.02 total 40.02 last 143.04) - open/close 7670 (time 1717018252.02 total 50.03 last 141.86) - open/close 9249 (time 1717018262.02 total 60.03 last 157.89) - open/close 10000 (time 1717018266.61 total 64.61 last 163.78) - open/close 11709 (time 1717018276.62 total 74.62 last 170.82) - open/close 13374 (time 1717018286.62 total 84.62 last 166.41) - open/close 14965 (time 1717018296.62 total 94.63 last 159.07) - open/close 16369 (time 1717018306.62 total 104.63 last 140.38) - open/close 17929 (time 1717018316.63 total 114.63 last 155.96) - open/close 19458 (time 1717018326.63 total 124.63 last 152.85) - open/close 20000 (time 1717018330.55 total 128.55 last 138.24) - open/close 21476 (time 1717018340.55 total 138.55 last 147.60) - open/close 22990 (time 1717018350.55 total 148.56 last 151.36) - open/close 24538 (time 1717018360.56 total 158.56 last 154.75) - open/close 26081 (time 1717018370.56 total 168.56 last 154.27) - open/close 27665 (time 1717018380.56 total 178.57 last 158.32) - open/close 29250 (time 1717018390.57 total 188.57 last 158.47) - open/close 30000 (time 1717018395.46 total 193.46 last 153.24) - open/close 31513 (time 1717018405.46 total 203.47 last 151.27) - open/close 32944 (time 1717018415.46 total 213.47 last 143.09) - open/close 34381 (time 1717018425.47 total 223.47 last 143.67) - open/close 35913 (time 1717018435.47 total 233.47 last 153.13) - open/close 37426 (time 1717018445.47 total 243.48 last 151.24) - open/close 38944 (time 1717018455.48 total 253.48 last 151.75) - open/close 40000 (time 1717018462.33 total 260.34 last 154.01) - open/close 41442 (time 1717018472.34 total 270.35 last 144.07) - open/close 43109 (time 1717018482.34 total 280.35 last 166.69) - open/close 44894 (time 1717018492.35 total 290.35 last 178.45) - open/close 46425 (time 1717018502.35 total 300.36 last 153.00) - open/close 48142 (time 1717018512.35 total 310.36 last 171.70) - open/close 49675 (time 1717018522.35 total 320.36 last 153.28) open(/mnt/lustre/d106.conf-sanity/f-49698) error: No space left on device total: 49698 open/close in 320.54 seconds: 155.05 ops/second - /unlink 2624 (time 1717018534.05 total 10.00 last 262.36) - /unlink 5375 (time 1717018544.05 total 20.00 last 275.08) - /unlink 8134 (time 1717018554.06 total 30.01 last 275.76) - /unlink 10000 (time 1717018561.01 total 36.96 last 268.55) - /unlink 12393 (time 1717018571.01 total 46.96 last 239.26) - /unlink 14837 (time 1717018581.01 total 56.96 last 244.40) - /unlink 17232 (time 1717018591.01 total 66.96 last 239.49) - /unlink 19646 (time 1717018601.01 total 76.96 last 241.29) - /unlink 21967 (time 1717018611.01 total 86.96 last 232.09) - /unlink 24694 (time 1717018621.02 total 96.97 last 272.62) - /unlink 27260 (time 1717018631.02 total 106.97 last 256.57) - /unlink 30000 (time 1717018640.20 total 116.15 last 298.28) - /unlink 32668 (time 1717018650.21 total 126.16 last 266.62) - /unlink 35376 (time 1717018660.21 total 136.16 last 270.76) - /unlink 37991 (time 1717018670.22 total 146.17 last 261.40) - /unlink 40000 (time 1717018678.53 total 154.48 last 241.59) - /unlink 42560 (time 1717018688.53 total 164.48 last 255.98) - /unlink 45214 (time 1717018698.54 total 174.49 last 265.22) - /unlink 47846 (time 1717018708.54 total 184.49 last 263.15) unlink(/mnt/lustre/d106.conf-sanity/f-49698) error: No such file or directory total: 49698 /unlink in 191.31 seconds: 259.78 ops/second - open/close 1519 (time 1717018727.12 total 10.01 last 151.77) - open/close 3080 (time 1717018737.12 total 20.01 last 156.10) - open/close 4810 (time 1717018747.13 total 30.01 last 172.91) - open/close 6426 (time 1717018757.13 total 40.01 last 161.58) - open/close 8126 (time 1717018767.13 total 50.02 last 170.00) - open/close 9892 (time 1717018777.13 total 60.02 last 176.58) - open/close 11702 (time 1717018787.14 total 70.02 last 180.90) - open/close 13281 (time 1717018797.14 total 80.03 last 157.81) - open/close 15140 (time 1717018807.14 total 90.03 last 185.85) - open/close 16749 (time 1717018817.14 total 100.03 last 160.89) - open/close 18245 (time 1717018827.15 total 110.03 last 149.56) - open/close 19920 (time 1717018837.15 total 120.03 last 167.49) - open/close 21805 (time 1717018847.15 total 130.04 last 188.38) - open/close 23478 (time 1717018857.16 total 140.04 last 167.25) - open/close 25080 (time 1717018867.16 total 150.05 last 160.11) - open/close 26654 (time 1717018877.17 total 160.05 last 157.36) - open/close 28293 (time 1717018887.17 total 170.06 last 163.80) - open/close 29901 (time 1717018897.17 total 180.06 last 160.77) - open/close 31394 (time 1717018907.17 total 190.06 last 149.28) - open/close 32971 (time 1717018917.18 total 200.06 last 157.67) - open/close 34538 (time 1717018927.18 total 210.07 last 156.61) - open/close 36051 (time 1717018937.19 total 220.07 last 151.23) - open/close 37432 (time 1717018947.19 total 230.07 last 138.06) - open/close 38853 (time 1717018957.19 total 240.08 last 142.08) - open/close 40000 (time 1717018964.79 total 247.67 last 150.98) - open/close 41392 (time 1717018974.79 total 257.67 last 139.19) - open/close 42829 (time 1717018984.79 total 267.68 last 143.66) - open/close 44235 (time 1717018994.80 total 277.68 last 140.48) - open/close 45588 (time 1717019004.80 total 287.69 last 135.26) - open/close 46951 (time 1717019014.81 total 297.69 last 136.26) - open/close 48310 (time 1717019024.81 total 307.69 last 135.88) open(/mnt/lustre/d106.conf-sanity/f-49698) error: No space left on device total: 49698 open/close in 317.45 seconds: 156.56 ops/second - /unlink 2593 (time 1717019046.00 total 10.01 last 259.17) - /unlink 5155 (time 1717019056.01 total 20.01 last 256.11) - /unlink 7703 (time 1717019066.01 total 30.01 last 254.77) - /unlink 10000 (time 1717019074.87 total 38.87 last 259.26) - /unlink 12639 (time 1717019084.87 total 48.87 last 263.84) - /unlink 15266 (time 1717019094.87 total 58.87 last 262.64) - /unlink 17824 (time 1717019104.88 total 68.88 last 255.75) - /unlink 20000 (time 1717019112.64 total 76.64 last 280.38) - /unlink 22793 (time 1717019122.64 total 86.64 last 279.25) - /unlink 25137 (time 1717019132.64 total 96.64 last 234.37) - /unlink 27435 (time 1717019142.64 total 106.64 last 229.69) - /unlink 29821 (time 1717019152.65 total 116.65 last 238.50) - /unlink 32378 (time 1717019162.65 total 126.65 last 255.61) - /unlink 34947 (time 1717019172.65 total 136.66 last 256.84) - /unlink 37705 (time 1717019182.66 total 146.66 last 275.77) - /unlink 40000 (time 1717019189.50 total 153.50 last 335.28) - /unlink 43197 (time 1717019199.50 total 163.50 last 319.58) - /unlink 46264 (time 1717019209.51 total 173.51 last 306.67) - /unlink 49185 (time 1717019219.51 total 183.51 last 292.04) unlink(/mnt/lustre/d106.conf-sanity/f-49698) error: No such file or directory total: 49698 /unlink in 185.54 seconds: 267.85 ops/second - open/close 1455 (time 1717019233.26 total 10.00 last 145.49) - open/close 2926 (time 1717019243.27 total 20.00 last 147.05) - open/close 4450 (time 1717019253.27 total 30.01 last 152.35) - open/close 5877 (time 1717019263.27 total 40.01 last 142.64) - open/close 7248 (time 1717019273.28 total 50.02 last 136.98) - open/close 8673 (time 1717019283.28 total 60.02 last 142.46) - open/close 10000 (time 1717019292.55 total 69.29 last 143.22) - open/close 11430 (time 1717019302.55 total 79.29 last 142.96) - open/close 12935 (time 1717019312.55 total 89.29 last 150.47) - open/close 14375 (time 1717019322.56 total 99.29 last 143.99) - open/close 15938 (time 1717019332.56 total 109.30 last 156.27) - open/close 17584 (time 1717019342.56 total 119.30 last 164.57) - open/close 19166 (time 1717019352.56 total 129.30 last 158.11) - open/close 20000 (time 1717019358.13 total 134.87 last 149.92) - open/close 21475 (time 1717019368.13 total 144.87 last 147.50) - open/close 22817 (time 1717019378.13 total 154.87 last 134.16) - open/close 24272 (time 1717019388.13 total 164.87 last 145.46) - open/close 25804 (time 1717019398.14 total 174.88 last 153.16) - open/close 27369 (time 1717019408.14 total 184.88 last 156.49) - open/close 28983 (time 1717019418.14 total 194.88 last 161.39) - open/close 30000 (time 1717019425.01 total 201.74 last 148.09) - open/close 31379 (time 1717019435.01 total 211.75 last 137.80) - open/close 33255 (time 1717019445.01 total 221.75 last 187.58) - open/close 35048 (time 1717019455.02 total 231.76 last 179.19) - open/close 36911 (time 1717019465.02 total 241.76 last 186.29) - open/close 38692 (time 1717019475.02 total 251.76 last 178.03) - open/close 40000 (time 1717019482.66 total 259.40 last 171.24) - open/close 41780 (time 1717019492.66 total 269.40 last 177.98) - open/close 43402 (time 1717019502.67 total 279.40 last 162.17) - open/close 44865 (time 1717019512.67 total 289.41 last 146.18) - open/close 46264 (time 1717019522.68 total 299.42 last 139.84) - open/close 47815 (time 1717019532.68 total 309.42 last 155.04) - open/close 49303 (time 1717019542.69 total 319.43 last 148.71) open(/mnt/lustre/d106.conf-sanity/f-49698) error: No space left on device total: 49698 open/close in 321.92 seconds: 154.38 ops/second - /unlink 2597 (time 1717019556.49 total 10.00 last 259.68) - /unlink 4977 (time 1717019566.49 total 20.00 last 237.95) - /unlink 7230 (time 1717019576.49 total 30.00 last 225.25) - /unlink 10000 (time 1717019585.43 total 38.95 last 309.82) - /unlink 12356 (time 1717019595.44 total 48.95 last 235.53) - /unlink 15202 (time 1717019605.44 total 58.95 last 284.53) - /unlink 18113 (time 1717019615.44 total 68.95 last 291.07) - /unlink 20000 (time 1717019622.53 total 76.04 last 266.23) - /unlink 22442 (time 1717019632.53 total 86.04 last 244.13) - /unlink 25632 (time 1717019642.53 total 96.04 last 318.98) - /unlink 28696 (time 1717019652.53 total 106.05 last 306.34) - /unlink 30000 (time 1717019656.66 total 110.17 last 315.76) - /unlink 32968 (time 1717019666.67 total 120.18 last 296.73) - /unlink 35327 (time 1717019676.67 total 130.18 last 235.80) - /unlink 37718 (time 1717019686.67 total 140.18 last 239.09) - /unlink 40000 (time 1717019695.21 total 148.72 last 267.19) - /unlink 42780 (time 1717019705.25 total 158.76 last 276.86) - /unlink 46488 (time 1717019715.25 total 168.77 last 370.71) - /unlink 49391 (time 1717019725.26 total 178.77 last 290.27) unlink(/mnt/lustre/d106.conf-sanity/f-49698) error: No such file or directory total: 49698 /unlink in 179.62 seconds: 276.69 ops/second - open/close 1453 (time 1717019737.53 total 10.00 last 145.30) - open/close 2798 (time 1717019747.54 total 20.00 last 134.45) - open/close 4169 (time 1717019757.54 total 30.01 last 137.03) - open/close 5574 (time 1717019767.55 total 40.01 last 140.44) - open/close 7060 (time 1717019777.55 total 50.01 last 148.59) - open/close 8571 (time 1717019787.55 total 60.01 last 151.08) - open/close 9936 (time 1717019797.56 total 70.02 last 136.40) - open/close 11388 (time 1717019807.56 total 80.03 last 145.13) - open/close 12809 (time 1717019817.56 total 90.03 last 142.07) - open/close 14448 (time 1717019827.58 total 100.04 last 163.66) - open/close 15994 (time 1717019837.58 total 110.04 last 154.59) - open/close 17661 (time 1717019847.58 total 120.05 last 166.64) - open/close 19505 (time 1717019857.58 total 130.05 last 184.37) - open/close 20000 (time 1717019860.70 total 133.17 last 158.85) - open/close 21697 (time 1717019870.70 total 143.17 last 169.65) - open/close 23645 (time 1717019880.70 total 153.17 last 194.78) - open/close 25450 (time 1717019890.71 total 163.18 last 180.38) - open/close 27075 (time 1717019900.71 total 173.18 last 162.49) - open/close 28677 (time 1717019910.71 total 183.18 last 160.14) - open/close 30000 (time 1717019918.32 total 190.78 last 174.04) - open/close 32016 (time 1717019928.32 total 200.79 last 201.53) - open/close 33828 (time 1717019938.33 total 210.79 last 181.08) - open/close 35367 (time 1717019948.33 total 220.79 last 153.90) - open/close 36930 (time 1717019958.34 total 230.80 last 156.17) - open/close 38583 (time 1717019968.34 total 240.81 last 165.20) - open/close 40000 (time 1717019976.37 total 248.83 last 176.59) - open/close 41713 (time 1717019986.37 total 258.83 last 171.26) - open/close 43338 (time 1717019996.37 total 268.84 last 162.47) - open/close 44694 (time 1717020006.37 total 278.84 last 135.58) - open/close 46090 (time 1717020016.38 total 288.84 last 139.50) - open/close 47485 (time 1717020026.38 total 298.85 last 139.44) - open/close 49049 (time 1717020036.39 total 308.85 last 156.35) open(/mnt/lustre/d106.conf-sanity/f-49590) error: No space left on device total: 49590 open/close in 312.57 seconds: 158.65 ops/second - /unlink 2635 (time 1717020051.75 total 10.00 last 263.38) - /unlink 5223 (time 1717020061.75 total 20.01 last 258.74) - /unlink 8663 (time 1717020071.75 total 30.01 last 343.99) - /unlink 10000 (time 1717020075.52 total 33.77 last 354.99) - /unlink 12826 (time 1717020085.52 total 43.78 last 282.53) - /unlink 15467 (time 1717020095.52 total 53.78 last 264.06) - /unlink 18433 (time 1717020105.52 total 63.78 last 296.54) - /unlink 20000 (time 1717020111.99 total 70.25 last 242.35) - /unlink 22929 (time 1717020121.99 total 80.25 last 292.86) - /unlink 25597 (time 1717020131.99 total 90.25 last 266.76) - /unlink 28133 (time 1717020142.00 total 100.25 last 253.50) - /unlink 30000 (time 1717020150.19 total 108.45 last 227.83) - /unlink 32191 (time 1717020160.20 total 118.45 last 218.97) - /unlink 34595 (time 1717020170.20 total 128.45 last 240.35) - /unlink 37059 (time 1717020180.20 total 138.46 last 246.36) - /unlink 39544 (time 1717020190.20 total 148.46 last 248.49) - /unlink 42249 (time 1717020200.20 total 158.46 last 270.46) - /unlink 45338 (time 1717020210.20 total 168.46 last 308.88) - /unlink 47910 (time 1717020220.20 total 178.46 last 257.16) unlink(/mnt/lustre/d106.conf-sanity/f-49590) error: No such file or directory total: 49590 /unlink in 185.42 seconds: 267.44 ops/second - open/close 1490 (time 1717020238.84 total 10.00 last 148.95) - open/close 2887 (time 1717020248.84 total 20.01 last 139.66) - open/close 4319 (time 1717020258.85 total 30.01 last 143.16) - open/close 5750 (time 1717020268.85 total 40.02 last 143.01) - open/close 7265 (time 1717020278.85 total 50.02 last 151.49) - open/close 8704 (time 1717020288.86 total 60.02 last 143.85) - open/close 10000 (time 1717020297.48 total 68.64 last 150.37) - open/close 11435 (time 1717020307.48 total 78.64 last 143.43) - open/close 12927 (time 1717020317.48 total 88.64 last 149.17) - open/close 14459 (time 1717020327.48 total 98.65 last 153.17) - open/close 15946 (time 1717020337.49 total 108.65 last 148.65) - open/close 17393 (time 1717020347.49 total 118.65 last 144.66) - open/close 18740 (time 1717020357.49 total 128.65 last 134.70) - open/close 20000 (time 1717020366.02 total 137.18 last 147.70) - open/close 21419 (time 1717020376.03 total 147.19 last 141.81) - open/close 22850 (time 1717020386.03 total 157.19 last 143.04) - open/close 24213 (time 1717020396.03 total 167.19 last 136.29) - open/close 25676 (time 1717020406.03 total 177.20 last 146.27) - open/close 27036 (time 1717020416.04 total 187.20 last 135.98) - open/close 28278 (time 1717020426.04 total 197.20 last 124.14) - open/close 29554 (time 1717020436.05 total 207.21 last 127.53) - open/close 30000 (time 1717020439.38 total 210.54 last 133.78) - open/close 31304 (time 1717020449.39 total 220.55 last 130.31) - open/close 32544 (time 1717020459.39 total 230.55 last 123.95) - open/close 33900 (time 1717020469.40 total 240.56 last 135.54) - open/close 35195 (time 1717020479.40 total 250.56 last 129.47) - open/close 36567 (time 1717020489.40 total 260.56 last 137.13) - open/close 37919 (time 1717020499.40 total 270.57 last 135.17) - open/close 39254 (time 1717020509.41 total 280.57 last 133.44) - open/close 40000 (time 1717020515.25 total 286.42 last 127.63) - open/close 41333 (time 1717020525.26 total 296.42 last 133.25) - open/close 42752 (time 1717020535.26 total 306.43 last 141.80) - open/close 44233 (time 1717020545.27 total 316.44 last 147.95) - open/close 45698 (time 1717020555.28 total 326.44 last 146.46) - open/close 47091 (time 1717020565.28 total 336.45 last 139.20) - open/close 48390 (time 1717020575.29 total 346.45 last 129.84) - open/close 49540 (time 1717020585.30 total 356.46 last 114.92) open(/mnt/lustre/d106.conf-sanity/f-49590) error: No space left on device total: 49590 open/close in 356.95 seconds: 138.93 ops/second - /unlink 1969 (time 1717020597.54 total 10.00 last 196.82) - /unlink 3838 (time 1717020607.54 total 20.01 last 186.86) - /unlink 5576 (time 1717020617.56 total 30.02 last 173.62) - /unlink 7298 (time 1717020627.56 total 40.02 last 172.20) - /unlink 9481 (time 1717020637.56 total 50.02 last 218.18) - /unlink 10000 (time 1717020639.75 total 52.21 last 236.94) - /unlink 12439 (time 1717020649.75 total 62.21 last 243.88) - /unlink 14715 (time 1717020659.75 total 72.22 last 227.55) - /unlink 16970 (time 1717020669.76 total 82.22 last 225.44) - /unlink 19227 (time 1717020679.76 total 92.22 last 225.66) - /unlink 20000 (time 1717020682.75 total 95.22 last 258.03) - /unlink 22500 (time 1717020692.76 total 105.22 last 249.91) - /unlink 25061 (time 1717020702.76 total 115.22 last 256.09) - /unlink 27286 (time 1717020712.76 total 125.22 last 222.43) - /unlink 29418 (time 1717020722.77 total 135.23 last 213.12) - /unlink 32204 (time 1717020732.77 total 145.23 last 278.49) - /unlink 34586 (time 1717020742.77 total 155.23 last 238.18) - /unlink 36923 (time 1717020752.77 total 165.23 last 233.69) - /unlink 39095 (time 1717020762.77 total 175.24 last 217.12) - /unlink 40000 (time 1717020766.64 total 179.11 last 233.88) - /unlink 42532 (time 1717020776.65 total 189.11 last 253.13) - /unlink 44921 (time 1717020786.65 total 199.11 last 238.87) - /unlink 47228 (time 1717020796.65 total 209.11 last 230.66) unlink(/mnt/lustre/d106.conf-sanity/f-49590) error: No such file or directory total: 49590 /unlink in 219.11 seconds: 226.33 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg128-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg128-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg128-server stop mds service on oleg128-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg128-server stop mds service on oleg128-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg128-server LNET ready to unload unloading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing unload_modules_local oleg128-server: LNET ready to unload modules unloaded. pdsh@oleg128-client: oleg128-client: ssh exited with exit code 2 pdsh@oleg128-client: oleg128-server: ssh exited with exit code 2 pdsh@oleg128-client: oleg128-client: ssh exited with exit code 2 pdsh@oleg128-client: oleg128-server: ssh exited with exit code 2 PASS 106 (2699s) Stopping clients: oleg128-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg128-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg128-client: oleg128-server: ssh exited with exit code 2 oleg128-server: oleg128-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: detected 4 online CPUs by sysfs oleg128-server: Force libcfs to create 2 CPU partitions oleg128-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg128-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 2838 sec ========== 18:14:40 (1717020880)