== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 05:40:04 (1787391604) Stopping clients: oleg105-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg105-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg105-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg105-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg105-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg105-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg105-server oleg105-server: oleg105-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg105-server' oleg105-server: oleg105-server.virtnet: executing load_modules_local oleg105-server: Loading modules from /home/green/git/lustre-release/lustre oleg105-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg105-server: detected 4 online CPUs by sysfs oleg105-server: MODOPTS_LIBCFS= oleg105-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg105-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg105-server: oleg105-server.virtnet: executing set_default_debug -1 all pdsh@oleg105-client: oleg105-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg105-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg105-server: oleg105-server.virtnet: executing set_default_debug -1 all pdsh@oleg105-client: oleg105-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg105-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg105-server: oleg105-server.virtnet: executing set_default_debug -1 all pdsh@oleg105-client: oleg105-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg105-client.virtnet: -o user_xattr,flock 192.168.201.105@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 698 (time 1787391848.95 total 10.01 last 69.73) - open/close 1476 (time 1787391858.96 total 20.01 last 77.79) - open/close 2221 (time 1787391868.97 total 30.02 last 74.43) - open/close 2866 (time 1787391878.97 total 40.02 last 64.49) - open/close 3581 (time 1787391888.97 total 50.03 last 71.45) - open/close 4434 (time 1787391898.99 total 60.04 last 85.18) total: 4500 open/close in 61.07 seconds: 73.68 ops/second - /unlink 1518 (time 1787391914.84 total 10.00 last 151.79) - /unlink 3054 (time 1787391924.84 total 20.00 last 153.57) total: 4500 /unlink in 29.88 seconds: 150.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 832 (time 1787391949.36 total 10.02 last 83.08) - open/close 1572 (time 1787391959.36 total 20.02 last 73.98) - open/close 2323 (time 1787391969.36 total 30.02 last 75.05) - open/close 3129 (time 1787391979.37 total 40.03 last 80.58) - open/close 3769 (time 1787391989.38 total 50.04 last 63.94) - open/close 4450 (time 1787391999.40 total 60.06 last 67.95) total: 4500 open/close in 60.62 seconds: 74.23 ops/second - /unlink 1359 (time 1787392015.51 total 10.01 last 135.83) - /unlink 2644 (time 1787392025.53 total 20.02 last 128.36) - /unlink 4016 (time 1787392035.53 total 30.02 last 137.19) total: 4500 /unlink in 33.17 seconds: 135.67 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 824 (time 1787392053.79 total 10.01 last 82.35) - open/close 1571 (time 1787392063.81 total 20.02 last 74.58) - open/close 2403 (time 1787392073.81 total 30.02 last 83.19) - open/close 3108 (time 1787392083.81 total 40.03 last 70.45) - open/close 3795 (time 1787392093.82 total 50.04 last 68.63) total: 4500 open/close in 58.10 seconds: 77.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1557 (time 1787392116.77 total 10.00 last 155.66) - /unlink 3059 (time 1787392126.77 total 20.01 last 150.16) total: 4500 /unlink in 29.27 seconds: 153.76 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 692 (time 1787392150.92 total 10.03 last 69.03) - open/close 1325 (time 1787392160.93 total 20.03 last 63.29) - open/close 2156 (time 1787392170.94 total 30.04 last 83.01) - open/close 2932 (time 1787392180.95 total 40.05 last 77.48) - open/close 3746 (time 1787392190.95 total 50.05 last 81.39) - open/close 4493 (time 1787392200.96 total 60.06 last 74.67) total: 4500 open/close in 60.11 seconds: 74.86 ops/second - /unlink 1202 (time 1787392216.42 total 10.01 last 120.07) - /unlink 2478 (time 1787392226.42 total 20.01 last 127.59) - /unlink 3819 (time 1787392236.42 total 30.02 last 134.05) total: 4500 /unlink in 35.03 seconds: 128.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 821 (time 1787392255.77 total 10.00 last 82.06) - open/close 1707 (time 1787392265.78 total 20.01 last 88.55) - open/close 2510 (time 1787392275.80 total 30.03 last 80.12) - open/close 3436 (time 1787392285.81 total 40.04 last 92.51) - open/close 4270 (time 1787392295.81 total 50.05 last 83.36) total: 4500 open/close in 52.81 seconds: 85.22 ops/second - /unlink 1564 (time 1787392313.41 total 10.00 last 156.40) - /unlink 3044 (time 1787392323.41 total 20.00 last 147.94) total: 4500 /unlink in 29.71 seconds: 151.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 679 (time 1787392348.64 total 10.01 last 67.86) - open/close 1384 (time 1787392358.65 total 20.01 last 70.45) - open/close 2107 (time 1787392368.66 total 30.03 last 72.21) - open/close 2635 (time 1787392378.67 total 40.03 last 52.75) - open/close 3014 (time 1787392388.69 total 50.05 last 37.83) - open/close 3572 (time 1787392398.71 total 60.07 last 55.69) - open/close 4196 (time 1787392408.72 total 70.09 last 62.32) total: 4500 open/close in 74.35 seconds: 60.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1568 (time 1787392427.81 total 10.00 last 156.76) - /unlink 3113 (time 1787392437.81 total 20.00 last 154.47) total: 4500 /unlink in 28.91 seconds: 155.67 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 916 (time 1787392461.16 total 10.01 last 91.54) - open/close 1772 (time 1787392471.17 total 20.02 last 85.51) - open/close 2691 (time 1787392481.17 total 30.02 last 91.86) - open/close 3515 (time 1787392491.18 total 40.02 last 82.37) - open/close 4368 (time 1787392501.18 total 50.03 last 85.27) total: 4500 open/close in 51.69 seconds: 87.06 ops/second - /unlink 1545 (time 1787392517.25 total 10.00 last 154.46) - /unlink 3167 (time 1787392527.26 total 20.01 last 162.15) total: 4500 /unlink in 29.29 seconds: 153.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 872 (time 1787392551.52 total 10.00 last 87.17) - open/close 1728 (time 1787392561.54 total 20.02 last 85.42) - open/close 2472 (time 1787392571.54 total 30.03 last 74.38) - open/close 3295 (time 1787392581.55 total 40.03 last 82.24) - open/close 4034 (time 1787392591.56 total 50.04 last 73.84) total: 4500 open/close in 55.97 seconds: 80.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1734 (time 1787392611.66 total 10.00 last 173.35) - /unlink 3132 (time 1787392621.66 total 20.00 last 139.79) total: 4500 /unlink in 28.27 seconds: 159.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 775 (time 1787392644.69 total 10.00 last 77.48) - open/close 1697 (time 1787392654.69 total 20.00 last 92.20) - open/close 2401 (time 1787392664.69 total 30.01 last 70.35) - open/close 3159 (time 1787392674.70 total 40.01 last 75.77) - open/close 3905 (time 1787392684.70 total 50.01 last 74.60) total: 4500 open/close in 58.40 seconds: 77.06 ops/second - /unlink 1385 (time 1787392707.23 total 10.00 last 138.45) - /unlink 2745 (time 1787392717.23 total 20.01 last 135.96) - /unlink 3960 (time 1787392727.23 total 30.01 last 121.47) total: 4500 /unlink in 34.34 seconds: 131.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 583 (time 1787392746.34 total 10.01 last 58.26) - open/close 1328 (time 1787392756.36 total 20.03 last 74.31) - open/close 2070 (time 1787392766.37 total 30.04 last 74.14) - open/close 2749 (time 1787392776.37 total 40.04 last 67.89) - open/close 3622 (time 1787392786.38 total 50.05 last 87.22) - open/close 4335 (time 1787392796.39 total 60.06 last 71.27) total: 4500 open/close in 62.10 seconds: 72.47 ops/second - /unlink 1455 (time 1787392813.22 total 10.00 last 145.47) - /unlink 2840 (time 1787392823.22 total 20.01 last 138.45) - /unlink 4317 (time 1787392833.23 total 30.01 last 147.65) total: 4500 /unlink in 31.25 seconds: 144.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 728 (time 1787392849.31 total 10.00 last 72.77) - open/close 1527 (time 1787392859.33 total 20.03 last 79.70) - open/close 2260 (time 1787392869.34 total 30.04 last 73.24) - open/close 3084 (time 1787392879.34 total 40.04 last 82.39) - open/close 3885 (time 1787392889.34 total 50.04 last 80.09) total: 4500 open/close in 57.72 seconds: 77.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1544 (time 1787392911.05 total 10.00 last 154.40) - /unlink 3139 (time 1787392921.06 total 20.00 last 159.43) total: 4500 /unlink in 29.30 seconds: 153.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 769 (time 1787392945.05 total 10.01 last 76.85) - open/close 1660 (time 1787392955.05 total 20.01 last 89.08) - open/close 2501 (time 1787392965.06 total 30.02 last 84.00) - open/close 3379 (time 1787392975.07 total 40.03 last 87.73) - open/close 4156 (time 1787392985.07 total 50.03 last 77.68) total: 4500 open/close in 53.71 seconds: 83.78 ops/second - /unlink 1409 (time 1787393002.44 total 10.00 last 140.87) - /unlink 2702 (time 1787393012.48 total 20.04 last 128.81) - /unlink 4138 (time 1787393022.48 total 30.04 last 143.54) total: 4500 /unlink in 32.65 seconds: 137.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 868 (time 1787393039.59 total 10.00 last 86.77) - open/close 1761 (time 1787393049.59 total 20.01 last 89.27) - open/close 2590 (time 1787393059.59 total 30.01 last 82.89) - open/close 3418 (time 1787393069.60 total 40.02 last 82.72) - open/close 4359 (time 1787393079.64 total 50.05 last 93.79) total: 4500 open/close in 51.58 seconds: 87.25 ops/second - /unlink 1725 (time 1787393094.78 total 10.00 last 172.50) - /unlink 3006 (time 1787393104.79 total 20.01 last 127.98) - /unlink 4146 (time 1787393114.80 total 30.01 last 113.95) total: 4500 /unlink in 32.73 seconds: 137.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 900 (time 1787393131.98 total 10.01 last 89.91) - open/close 1805 (time 1787393141.98 total 20.02 last 90.45) - open/close 2558 (time 1787393151.99 total 30.02 last 75.25) - open/close 3461 (time 1787393161.99 total 40.03 last 90.26) - open/close 4283 (time 1787393172.00 total 50.04 last 82.12) total: 4500 open/close in 53.89 seconds: 83.50 ops/second - /unlink 1618 (time 1787393190.26 total 10.00 last 161.75) - /unlink 3217 (time 1787393200.26 total 20.01 last 159.85) total: 4500 /unlink in 26.65 seconds: 168.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 932 (time 1787393221.08 total 10.02 last 93.01) - open/close 1888 (time 1787393231.08 total 20.02 last 95.59) - open/close 2768 (time 1787393241.09 total 30.03 last 87.96) - open/close 3615 (time 1787393251.09 total 40.03 last 84.68) - open/close 4325 (time 1787393261.14 total 50.07 last 70.68) total: 4500 open/close in 52.53 seconds: 85.67 ops/second - /unlink 1303 (time 1787393277.42 total 10.01 last 130.16) - /unlink 2664 (time 1787393287.43 total 20.01 last 136.06) - /unlink 3851 (time 1787393297.43 total 30.02 last 118.64) total: 4500 /unlink in 34.89 seconds: 128.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 619 (time 1787393318.66 total 10.02 last 61.79) - open/close 1516 (time 1787393328.67 total 20.02 last 89.65) - open/close 2203 (time 1787393338.67 total 30.03 last 68.68) - open/close 2874 (time 1787393348.68 total 40.03 last 67.06) - open/close 3623 (time 1787393358.68 total 50.03 last 74.89) total: 4500 open/close in 59.99 seconds: 75.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1540 (time 1787393383.15 total 10.01 last 153.92) - /unlink 2943 (time 1787393393.16 total 20.01 last 140.26) total: 4500 /unlink in 29.88 seconds: 150.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 853 (time 1787393418.07 total 10.00 last 85.28) - open/close 1692 (time 1787393428.08 total 20.01 last 83.84) - open/close 2568 (time 1787393438.10 total 30.03 last 87.39) - open/close 3539 (time 1787393448.10 total 40.04 last 97.09) - open/close 4406 (time 1787393458.11 total 50.04 last 86.63) total: 4500 open/close in 51.20 seconds: 87.89 ops/second - /unlink 1613 (time 1787393473.75 total 10.00 last 161.29) - /unlink 3249 (time 1787393483.76 total 20.01 last 163.52) total: 4500 /unlink in 27.61 seconds: 162.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 763 (time 1787393506.17 total 10.00 last 76.28) - open/close 1598 (time 1787393516.18 total 20.01 last 83.42) - open/close 2500 (time 1787393526.18 total 30.02 last 90.16) - open/close 3335 (time 1787393536.19 total 40.03 last 83.43) - open/close 4201 (time 1787393546.19 total 50.03 last 86.58) total: 4500 open/close in 54.91 seconds: 81.95 ops/second - /unlink 1795 (time 1787393565.25 total 10.01 last 179.30) - /unlink 3200 (time 1787393575.26 total 20.02 last 140.35) total: 4500 /unlink in 28.99 seconds: 155.23 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11242: 94880 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user 'cl1': success lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user 'cl1': success Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg105-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg105-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg105-server stop mds service on oleg105-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg105-server stop mds service on oleg105-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg105-server unloading modules via unload_modules_local on: 'oleg105-server' oleg105-server: oleg105-server.virtnet: executing unload_modules_local oleg105-server: modules unloaded.