== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 02:26:58 (1787552818) Stopping clients: oleg621-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg621-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg621-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg621-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg621-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg621-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg621-server oleg621-server: oleg621-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg621-server' oleg621-server: oleg621-server.virtnet: executing load_modules_local oleg621-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg621-server: Loading modules from /home/green/git/lustre-release/lustre oleg621-server: detected 4 online CPUs by sysfs oleg621-server: MODOPTS_LIBCFS= oleg621-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg621-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg621-server: oleg621-server.virtnet: executing set_default_debug -1 all pdsh@oleg621-client: oleg621-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg621-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg621-server: oleg621-server.virtnet: executing set_default_debug -1 all pdsh@oleg621-client: oleg621-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg621-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg621-server: oleg621-server.virtnet: executing set_default_debug -1 all pdsh@oleg621-client: oleg621-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg621-client.virtnet: -o user_xattr,flock 192.168.206.121@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 715 (time 1787553034.35 total 10.01 last 71.43) - open/close 1338 (time 1787553044.36 total 20.02 last 62.25) - open/close 2005 (time 1787553054.36 total 30.02 last 66.68) - open/close 2627 (time 1787553064.36 total 40.02 last 62.20) - open/close 3292 (time 1787553074.38 total 50.03 last 66.42) - open/close 3908 (time 1787553084.38 total 60.04 last 61.58) total: 4500 open/close in 68.46 seconds: 65.73 ops/second - /unlink 1351 (time 1787553107.25 total 10.01 last 135.03) - /unlink 2639 (time 1787553117.26 total 20.01 last 128.75) - /unlink 4024 (time 1787553127.26 total 30.01 last 138.45) total: 4500 /unlink in 33.68 seconds: 133.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 693 (time 1787553145.22 total 10.02 last 69.17) - open/close 1445 (time 1787553155.23 total 20.02 last 75.16) - open/close 2208 (time 1787553165.23 total 30.03 last 76.28) - open/close 2959 (time 1787553175.23 total 40.03 last 75.09) - open/close 3641 (time 1787553185.24 total 50.04 last 68.12) - open/close 4359 (time 1787553195.25 total 60.04 last 71.78) total: 4500 open/close in 61.86 seconds: 72.74 ops/second - /unlink 1350 (time 1787553211.21 total 10.02 last 134.79) - /unlink 2690 (time 1787553221.21 total 20.02 last 133.97) - /unlink 4031 (time 1787553231.21 total 30.02 last 134.02) total: 4500 /unlink in 33.74 seconds: 133.36 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 654 (time 1787553250.06 total 10.01 last 65.31) - open/close 1304 (time 1787553260.07 total 20.03 last 64.91) - open/close 2076 (time 1787553270.08 total 30.04 last 77.14) - open/close 2856 (time 1787553280.08 total 40.04 last 78.00) - open/close 3576 (time 1787553290.10 total 50.05 last 71.88) - open/close 4302 (time 1787553300.10 total 60.06 last 72.58) total: 4500 open/close in 63.13 seconds: 71.29 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1527 (time 1787553318.03 total 10.00 last 152.65) - /unlink 2850 (time 1787553328.03 total 20.01 last 132.27) - /unlink 4121 (time 1787553338.03 total 30.01 last 127.10) total: 4500 /unlink in 33.12 seconds: 135.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 501 (time 1787553358.89 total 10.00 last 50.10) - open/close 1138 (time 1787553368.90 total 20.01 last 63.66) - open/close 1768 (time 1787553378.91 total 30.02 last 62.92) - open/close 2498 (time 1787553388.91 total 40.02 last 72.97) - open/close 3242 (time 1787553398.92 total 50.03 last 74.35) - open/close 3958 (time 1787553408.92 total 60.03 last 71.59) total: 4500 open/close in 67.25 seconds: 66.92 ops/second - /unlink 1429 (time 1787553431.18 total 10.01 last 142.83) - /unlink 2842 (time 1787553441.18 total 20.01 last 141.29) - /unlink 4262 (time 1787553451.19 total 30.01 last 141.95) total: 4500 /unlink in 31.70 seconds: 141.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 765 (time 1787553466.79 total 10.01 last 76.45) - open/close 1491 (time 1787553476.80 total 20.01 last 72.57) - open/close 2259 (time 1787553486.80 total 30.02 last 76.76) - open/close 2847 (time 1787553496.81 total 40.02 last 58.76) - open/close 3552 (time 1787553506.82 total 50.04 last 70.41) - open/close 4325 (time 1787553516.84 total 60.05 last 77.21) total: 4500 open/close in 62.16 seconds: 72.39 ops/second - /unlink 1465 (time 1787553532.18 total 10.00 last 146.49) - /unlink 2812 (time 1787553542.19 total 20.01 last 134.61) - /unlink 4029 (time 1787553552.19 total 30.01 last 121.68) total: 4500 /unlink in 33.21 seconds: 135.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 591 (time 1787553571.05 total 10.02 last 59.01) - open/close 1271 (time 1787553581.05 total 20.02 last 67.98) - open/close 2003 (time 1787553591.06 total 30.02 last 73.16) - open/close 2784 (time 1787553601.07 total 40.04 last 77.99) - open/close 3495 (time 1787553611.08 total 50.05 last 71.03) - open/close 4243 (time 1787553621.09 total 60.06 last 74.72) total: 4500 open/close in 63.44 seconds: 70.93 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1340 (time 1787553638.61 total 10.00 last 133.94) - /unlink 2447 (time 1787553648.62 total 20.01 last 110.61) - /unlink 3680 (time 1787553658.62 total 30.02 last 123.27) total: 4500 /unlink in 36.67 seconds: 122.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 714 (time 1787553681.53 total 10.00 last 71.39) - open/close 1456 (time 1787553691.53 total 20.00 last 74.19) - open/close 2166 (time 1787553701.54 total 30.01 last 70.93) - open/close 2912 (time 1787553711.54 total 40.01 last 74.58) - open/close 3641 (time 1787553721.55 total 50.03 last 72.82) - open/close 4425 (time 1787553731.56 total 60.03 last 78.34) total: 4500 open/close in 61.21 seconds: 73.52 ops/second - /unlink 1363 (time 1787553748.09 total 10.01 last 136.19) - /unlink 2732 (time 1787553758.10 total 20.01 last 136.85) - /unlink 4066 (time 1787553768.10 total 30.01 last 133.38) total: 4500 /unlink in 33.25 seconds: 135.36 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 755 (time 1787553785.70 total 10.02 last 75.36) - open/close 1505 (time 1787553795.70 total 20.02 last 74.98) - open/close 2188 (time 1787553805.71 total 30.03 last 68.21) - open/close 2912 (time 1787553815.72 total 40.04 last 72.37) - open/close 3632 (time 1787553825.72 total 50.04 last 71.99) - open/close 4392 (time 1787553835.72 total 60.04 last 75.99) total: 4500 open/close in 61.45 seconds: 73.23 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1440 (time 1787553850.51 total 10.00 last 143.94) - /unlink 2834 (time 1787553860.51 total 20.01 last 139.36) - /unlink 4295 (time 1787553870.52 total 30.01 last 146.04) total: 4500 /unlink in 31.35 seconds: 143.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 675 (time 1787553886.01 total 10.00 last 67.47) - open/close 1343 (time 1787553896.02 total 20.01 last 66.77) - open/close 2024 (time 1787553906.03 total 30.02 last 68.06) - open/close 2641 (time 1787553916.04 total 40.03 last 61.62) - open/close 3248 (time 1787553926.05 total 50.04 last 60.64) - open/close 4009 (time 1787553936.05 total 60.04 last 76.09) total: 4500 open/close in 67.43 seconds: 66.74 ops/second - /unlink 1410 (time 1787553957.54 total 10.00 last 140.94) - /unlink 2769 (time 1787553967.55 total 20.01 last 135.76) - /unlink 4164 (time 1787553977.55 total 30.02 last 139.47) total: 4500 /unlink in 32.45 seconds: 138.66 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 743 (time 1787553994.10 total 10.01 last 74.23) - open/close 1504 (time 1787554004.10 total 20.02 last 76.06) - open/close 2250 (time 1787554014.12 total 30.04 last 74.45) - open/close 2921 (time 1787554024.12 total 40.04 last 67.10) - open/close 3627 (time 1787554034.14 total 50.05 last 70.51) - open/close 4319 (time 1787554044.16 total 60.07 last 69.02) total: 4500 open/close in 62.65 seconds: 71.83 ops/second - /unlink 1300 (time 1787554061.05 total 10.00 last 129.95) - /unlink 2679 (time 1787554071.05 total 20.01 last 137.86) - /unlink 4035 (time 1787554081.05 total 30.01 last 135.58) total: 4500 /unlink in 34.02 seconds: 132.27 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 517 (time 1787554100.72 total 10.01 last 51.67) - open/close 1069 (time 1787554110.72 total 20.01 last 55.18) - open/close 1807 (time 1787554120.73 total 30.02 last 73.73) - open/close 2550 (time 1787554130.73 total 40.02 last 74.29) - open/close 3312 (time 1787554140.75 total 50.03 last 76.10) - open/close 4051 (time 1787554150.76 total 60.05 last 73.77) total: 4500 open/close in 66.89 seconds: 67.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1311 (time 1787554171.36 total 10.00 last 131.09) - /unlink 2514 (time 1787554181.37 total 20.00 last 120.27) - /unlink 3879 (time 1787554191.37 total 30.00 last 136.48) total: 4500 /unlink in 34.37 seconds: 130.93 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 731 (time 1787554210.29 total 10.01 last 73.06) - open/close 1536 (time 1787554220.30 total 20.01 last 80.46) - open/close 2304 (time 1787554230.30 total 30.02 last 76.76) - open/close 3117 (time 1787554240.31 total 40.02 last 81.26) - open/close 3875 (time 1787554250.31 total 50.03 last 75.76) total: 4500 open/close in 57.77 seconds: 77.89 ops/second - /unlink 1343 (time 1787554273.16 total 10.00 last 134.28) - /unlink 2684 (time 1787554283.16 total 20.01 last 134.06) - /unlink 4022 (time 1787554293.16 total 30.01 last 133.79) total: 4500 /unlink in 33.05 seconds: 136.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 803 (time 1787554309.48 total 10.00 last 80.29) - open/close 1520 (time 1787554319.49 total 20.01 last 71.64) - open/close 2290 (time 1787554329.49 total 30.01 last 76.96) - open/close 3096 (time 1787554339.50 total 40.02 last 80.58) - open/close 3869 (time 1787554349.50 total 50.02 last 77.25) total: 4500 open/close in 57.90 seconds: 77.72 ops/second - /unlink 1542 (time 1787554370.54 total 10.01 last 154.10) - /unlink 3060 (time 1787554380.54 total 20.01 last 151.76) total: 4500 /unlink in 29.71 seconds: 151.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 817 (time 1787554403.87 total 10.00 last 81.70) - open/close 1565 (time 1787554413.88 total 20.00 last 74.79) - open/close 2320 (time 1787554423.88 total 30.01 last 75.47) - open/close 3081 (time 1787554433.88 total 40.01 last 76.08) - open/close 3868 (time 1787554443.88 total 50.01 last 78.69) total: 4500 open/close in 58.79 seconds: 76.54 ops/second - /unlink 1426 (time 1787554467.46 total 10.00 last 142.55) - /unlink 2757 (time 1787554477.46 total 20.01 last 133.03) - /unlink 4193 (time 1787554487.47 total 30.02 last 143.49) total: 4500 /unlink in 32.27 seconds: 139.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 779 (time 1787554503.55 total 10.01 last 77.82) - open/close 1553 (time 1787554513.56 total 20.02 last 77.33) - open/close 2320 (time 1787554523.56 total 30.02 last 76.70) - open/close 3040 (time 1787554533.57 total 40.03 last 71.96) - open/close 3758 (time 1787554543.57 total 50.03 last 71.79) total: 4500 open/close in 59.92 seconds: 75.11 ops/second - /unlink 1439 (time 1787554567.97 total 10.00 last 143.87) - /unlink 2885 (time 1787554577.97 total 20.00 last 144.58) - /unlink 4378 (time 1787554587.98 total 30.01 last 149.22) total: 4500 /unlink in 31.02 seconds: 145.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 761 (time 1787554604.06 total 10.01 last 75.99) - open/close 1510 (time 1787554614.08 total 20.03 last 74.79) - open/close 2267 (time 1787554624.08 total 30.04 last 75.66) - open/close 3006 (time 1787554634.09 total 40.04 last 73.89) - open/close 3791 (time 1787554644.09 total 50.04 last 78.46) - open/close 4463 (time 1787554654.09 total 60.05 last 67.17) total: 4500 open/close in 60.50 seconds: 74.38 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1304 (time 1787554668.61 total 10.01 last 130.33) - /unlink 2704 (time 1787554678.61 total 20.01 last 139.98) - /unlink 3855 (time 1787554688.61 total 30.01 last 115.07) total: 4500 /unlink in 35.13 seconds: 128.08 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 621 (time 1787554709.14 total 10.00 last 62.08) - open/close 1305 (time 1787554719.16 total 20.02 last 68.30) - open/close 2005 (time 1787554729.16 total 30.02 last 69.96) - open/close 2728 (time 1787554739.16 total 40.02 last 72.29) - open/close 3410 (time 1787554749.17 total 50.03 last 68.14) - open/close 4094 (time 1787554759.18 total 60.04 last 68.32) total: 4500 open/close in 66.50 seconds: 67.67 ops/second - /unlink 1177 (time 1787554781.43 total 10.00 last 117.66) - /unlink 2263 (time 1787554791.43 total 20.00 last 108.58) - /unlink 3326 (time 1787554801.43 total 30.01 last 106.27) total: 4500 /unlink in 39.30 seconds: 114.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 726 (time 1787554825.23 total 10.01 last 72.53) - open/close 1449 (time 1787554835.23 total 20.02 last 72.24) - open/close 2203 (time 1787554845.23 total 30.02 last 75.40) - open/close 2958 (time 1787554855.24 total 40.02 last 75.46) - open/close 3606 (time 1787554865.25 total 50.03 last 64.76) - open/close 4348 (time 1787554875.26 total 60.04 last 74.09) total: 4500 open/close in 62.42 seconds: 72.09 ops/second - /unlink 1404 (time 1787554891.89 total 10.00 last 140.35) - /unlink 2666 (time 1787554901.89 total 20.00 last 126.19) - /unlink 3773 (time 1787554911.90 total 30.01 last 110.59) total: 4500 /unlink in 36.92 seconds: 121.90 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95313 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg621-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg621-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg621-server stop mds service on oleg621-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg621-server stop mds service on oleg621-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg621-server unloading modules via unload_modules_local on: 'oleg621-server' oleg621-server: oleg621-server.virtnet: executing unload_modules_local oleg621-server: modules unloaded.