== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 13:28:03 (1784050083) Stopping clients: oleg129-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg129-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg129-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg129-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg129-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg129-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg129-server oleg129-server: oleg129-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg129-server' oleg129-server: oleg129-server.virtnet: executing load_modules_local oleg129-server: Loading modules from /home/green/git/lustre-release/lustre oleg129-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg129-server: detected 4 online CPUs by sysfs oleg129-server: MODOPTS_LIBCFS= oleg129-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg129-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg129-server: oleg129-server.virtnet: executing set_default_debug -1 all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg129-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg129-server: oleg129-server.virtnet: executing set_default_debug -1 all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg129-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg129-server: oleg129-server.virtnet: executing set_default_debug -1 all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg129-client.virtnet: -o user_xattr,flock 192.168.201.129@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 748 (time 1784050286.74 total 10.02 last 74.66) - open/close 1416 (time 1784050296.75 total 20.03 last 66.75) - open/close 2075 (time 1784050306.76 total 30.03 last 65.85) - open/close 2766 (time 1784050316.80 total 40.07 last 68.82) - open/close 3447 (time 1784050326.81 total 50.08 last 68.04) - open/close 4192 (time 1784050336.84 total 60.11 last 74.29) total: 4500 open/close in 64.71 seconds: 69.55 ops/second - /unlink 1272 (time 1784050355.93 total 10.00 last 127.18) - /unlink 2750 (time 1784050365.93 total 20.01 last 147.72) - /unlink 4300 (time 1784050375.94 total 30.01 last 154.95) total: 4500 /unlink in 31.68 seconds: 142.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 774 (time 1784050392.34 total 10.01 last 77.35) - open/close 1404 (time 1784050402.35 total 20.02 last 62.94) - open/close 2104 (time 1784050412.35 total 30.02 last 69.99) - open/close 2871 (time 1784050422.36 total 40.02 last 76.67) - open/close 3590 (time 1784050432.36 total 50.02 last 71.88) - open/close 4334 (time 1784050442.37 total 60.04 last 74.29) total: 4500 open/close in 62.49 seconds: 72.01 ops/second - /unlink 1630 (time 1784050458.39 total 10.01 last 162.91) - /unlink 2972 (time 1784050468.39 total 20.01 last 134.19) - /unlink 4418 (time 1784050478.39 total 30.01 last 144.51) total: 4500 /unlink in 30.67 seconds: 146.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 726 (time 1784050493.92 total 10.01 last 72.49) - open/close 1491 (time 1784050503.94 total 20.04 last 76.34) - open/close 2300 (time 1784050513.97 total 30.06 last 80.70) - open/close 3063 (time 1784050523.97 total 40.06 last 76.28) - open/close 3681 (time 1784050533.98 total 50.07 last 61.74) - open/close 4165 (time 1784050544.00 total 60.09 last 48.31) total: 4500 open/close in 65.43 seconds: 68.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1461 (time 1784050564.80 total 10.01 last 146.02) - /unlink 2772 (time 1784050574.80 total 20.01 last 131.05) - /unlink 3994 (time 1784050584.80 total 30.01 last 122.17) total: 4500 /unlink in 33.37 seconds: 134.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 662 (time 1784050603.26 total 10.01 last 66.16) - open/close 1318 (time 1784050613.27 total 20.02 last 65.54) - open/close 2087 (time 1784050623.29 total 30.04 last 76.75) - open/close 2799 (time 1784050633.30 total 40.04 last 71.17) - open/close 3366 (time 1784050643.30 total 50.04 last 56.69) - open/close 3894 (time 1784050653.31 total 60.05 last 52.75) - open/close 4372 (time 1784050663.33 total 70.07 last 47.71) total: 4500 open/close in 72.79 seconds: 61.82 ops/second - /unlink 1264 (time 1784050681.44 total 10.00 last 126.39) - /unlink 2689 (time 1784050691.44 total 20.00 last 142.45) - /unlink 4095 (time 1784050701.45 total 30.01 last 140.49) total: 4500 /unlink in 33.21 seconds: 135.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 671 (time 1784050719.93 total 10.00 last 67.09) - open/close 1200 (time 1784050729.97 total 20.04 last 52.69) - open/close 1794 (time 1784050739.99 total 30.05 last 59.33) - open/close 2473 (time 1784050749.99 total 40.06 last 67.85) - open/close 3154 (time 1784050760.00 total 50.07 last 68.05) - open/close 3955 (time 1784050770.01 total 60.07 last 80.04) total: 4500 open/close in 67.01 seconds: 67.16 ops/second - /unlink 1104 (time 1784050791.91 total 10.01 last 110.33) - /unlink 2374 (time 1784050801.92 total 20.01 last 126.98) - /unlink 3766 (time 1784050811.92 total 30.02 last 139.09) total: 4500 /unlink in 35.36 seconds: 127.27 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 745 (time 1784050831.37 total 10.00 last 74.48) - open/close 1502 (time 1784050841.38 total 20.01 last 75.63) - open/close 2272 (time 1784050851.42 total 30.05 last 76.71) - open/close 2888 (time 1784050861.43 total 40.06 last 61.52) - open/close 3650 (time 1784050871.44 total 50.07 last 76.12) - open/close 4361 (time 1784050881.44 total 60.07 last 71.10) total: 4500 open/close in 62.03 seconds: 72.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1071 (time 1784050898.61 total 10.01 last 107.03) - /unlink 2283 (time 1784050908.62 total 20.01 last 121.18) - /unlink 3691 (time 1784050918.62 total 30.01 last 140.77) total: 4500 /unlink in 36.03 seconds: 124.90 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 744 (time 1784050939.19 total 10.02 last 74.26) - open/close 1548 (time 1784050949.20 total 20.03 last 80.33) - open/close 2252 (time 1784050959.21 total 30.04 last 70.33) - open/close 3022 (time 1784050969.23 total 40.05 last 76.87) - open/close 3791 (time 1784050979.24 total 50.06 last 76.82) total: 4500 open/close in 59.23 seconds: 75.98 ops/second - /unlink 1349 (time 1784051003.08 total 10.01 last 134.81) - /unlink 2771 (time 1784051013.09 total 20.01 last 142.16) - /unlink 4082 (time 1784051023.09 total 30.01 last 131.03) total: 4500 /unlink in 33.85 seconds: 132.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 571 (time 1784051043.73 total 10.02 last 56.97) - open/close 1184 (time 1784051053.73 total 20.03 last 61.24) - open/close 1873 (time 1784051063.74 total 30.04 last 68.86) - open/close 2624 (time 1784051073.75 total 40.04 last 75.06) - open/close 3348 (time 1784051083.76 total 50.05 last 72.32) - open/close 4015 (time 1784051093.78 total 60.07 last 66.58) total: 4500 open/close in 67.62 seconds: 66.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1409 (time 1784051116.01 total 10.01 last 140.78) - /unlink 2634 (time 1784051126.01 total 20.01 last 122.45) - /unlink 3991 (time 1784051136.01 total 30.01 last 135.69) total: 4500 /unlink in 33.55 seconds: 134.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 726 (time 1784051154.37 total 10.00 last 72.59) - open/close 1469 (time 1784051164.38 total 20.01 last 74.22) - open/close 2198 (time 1784051174.38 total 30.02 last 72.86) - open/close 2893 (time 1784051184.39 total 40.02 last 69.48) - open/close 3464 (time 1784051194.39 total 50.03 last 57.06) - open/close 4054 (time 1784051204.43 total 60.07 last 58.77) total: 4500 open/close in 66.72 seconds: 67.45 ops/second - /unlink 1324 (time 1784051226.10 total 10.00 last 132.35) - /unlink 2729 (time 1784051236.10 total 20.01 last 140.43) - /unlink 4066 (time 1784051246.11 total 30.01 last 133.65) total: 4500 /unlink in 33.78 seconds: 133.23 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 562 (time 1784051266.45 total 10.01 last 56.12) - open/close 1049 (time 1784051276.45 total 20.02 last 48.68) - open/close 1582 (time 1784051286.46 total 30.03 last 53.24) - open/close 2251 (time 1784051296.47 total 40.03 last 66.87) - open/close 3056 (time 1784051306.47 total 50.03 last 80.50) - open/close 3817 (time 1784051316.47 total 60.04 last 76.09) total: 4500 open/close in 68.05 seconds: 66.13 ops/second - /unlink 1121 (time 1784051339.38 total 10.00 last 112.10) - /unlink 2202 (time 1784051349.39 total 20.00 last 108.05) - /unlink 3525 (time 1784051359.39 total 30.01 last 132.22) total: 4500 /unlink in 38.38 seconds: 117.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 778 (time 1784051383.90 total 10.01 last 77.73) - open/close 1500 (time 1784051393.91 total 20.01 last 72.16) - open/close 2216 (time 1784051403.92 total 30.03 last 71.52) - open/close 2913 (time 1784051413.93 total 40.03 last 69.64) - open/close 3691 (time 1784051423.94 total 50.04 last 77.74) - open/close 4448 (time 1784051433.95 total 60.05 last 75.63) total: 4500 open/close in 60.69 seconds: 74.15 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1318 (time 1784051450.07 total 10.02 last 131.56) - /unlink 2536 (time 1784051460.08 total 20.03 last 121.69) - /unlink 3884 (time 1784051470.09 total 30.03 last 134.74) total: 4500 /unlink in 35.25 seconds: 127.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 676 (time 1784051492.34 total 10.01 last 67.55) - open/close 1399 (time 1784051502.35 total 20.02 last 72.24) - open/close 2081 (time 1784051512.36 total 30.02 last 68.14) - open/close 2737 (time 1784051522.36 total 40.03 last 65.56) - open/close 3441 (time 1784051532.39 total 50.06 last 70.22) - open/close 4171 (time 1784051542.39 total 60.06 last 72.97) total: 4500 open/close in 63.86 seconds: 70.46 ops/second - /unlink 1337 (time 1784051560.35 total 10.01 last 133.59) - /unlink 2626 (time 1784051570.35 total 20.01 last 128.87) - /unlink 3913 (time 1784051580.36 total 30.01 last 128.67) total: 4500 /unlink in 34.33 seconds: 131.08 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 788 (time 1784051599.75 total 10.00 last 78.78) - open/close 1567 (time 1784051609.75 total 20.00 last 77.89) - open/close 2352 (time 1784051619.76 total 30.01 last 78.44) - open/close 3011 (time 1784051629.77 total 40.02 last 65.85) - open/close 3579 (time 1784051639.77 total 50.02 last 56.80) - open/close 4228 (time 1784051649.79 total 60.04 last 64.77) total: 4500 open/close in 64.52 seconds: 69.74 ops/second - /unlink 1550 (time 1784051669.21 total 10.01 last 154.84) - /unlink 2515 (time 1784051679.21 total 20.01 last 96.49) - /unlink 3668 (time 1784051689.22 total 30.01 last 115.29) total: 4500 /unlink in 37.00 seconds: 121.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 590 (time 1784051710.62 total 10.02 last 58.87) - open/close 1330 (time 1784051720.62 total 20.03 last 73.96) - open/close 1986 (time 1784051730.62 total 30.03 last 65.60) - open/close 2724 (time 1784051740.63 total 40.04 last 73.73) - open/close 3503 (time 1784051750.63 total 50.04 last 77.89) - open/close 4211 (time 1784051760.64 total 60.04 last 70.78) total: 4500 open/close in 63.98 seconds: 70.34 ops/second - /unlink 1521 (time 1784051778.65 total 10.00 last 152.06) - /unlink 3010 (time 1784051788.65 total 20.00 last 148.90) - /unlink 4233 (time 1784051798.65 total 30.01 last 122.24) total: 4500 /unlink in 32.43 seconds: 138.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 809 (time 1784051816.11 total 10.00 last 80.88) - open/close 1527 (time 1784051826.12 total 20.01 last 71.74) - open/close 2369 (time 1784051836.12 total 30.01 last 84.18) - open/close 3200 (time 1784051846.13 total 40.02 last 83.04) - open/close 3946 (time 1784051856.14 total 50.03 last 74.53) total: 4500 open/close in 57.63 seconds: 78.09 ops/second - /unlink 1468 (time 1784051878.25 total 10.00 last 146.75) - /unlink 2849 (time 1784051888.25 total 20.01 last 138.04) - /unlink 4443 (time 1784051898.26 total 30.02 last 159.27) total: 4500 /unlink in 30.63 seconds: 146.94 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 767 (time 1784051913.36 total 10.02 last 76.55) - open/close 1468 (time 1784051923.37 total 20.03 last 70.03) - open/close 2218 (time 1784051933.39 total 30.05 last 74.84) - open/close 2934 (time 1784051943.40 total 40.06 last 71.55) - open/close 3579 (time 1784051953.41 total 50.07 last 64.41) - open/close 4268 (time 1784051963.41 total 60.08 last 68.87) total: 4500 open/close in 63.75 seconds: 70.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1095 (time 1784051982.37 total 10.00 last 109.46) - /unlink 2231 (time 1784051992.37 total 20.01 last 113.52) - /unlink 3477 (time 1784052002.37 total 30.01 last 124.59) total: 4500 /unlink in 37.73 seconds: 119.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 787 (time 1784052025.75 total 10.03 last 78.49) - open/close 1501 (time 1784052035.76 total 20.04 last 71.33) - open/close 2205 (time 1784052045.76 total 30.04 last 70.39) - open/close 2911 (time 1784052055.77 total 40.04 last 70.57) - open/close 3574 (time 1784052065.78 total 50.05 last 66.24) - open/close 4240 (time 1784052075.78 total 60.06 last 66.55) total: 4500 open/close in 63.83 seconds: 70.50 ops/second - /unlink 1234 (time 1784052094.84 total 10.01 last 123.34) - /unlink 2428 (time 1784052104.84 total 20.01 last 119.35) - /unlink 3498 (time 1784052114.85 total 30.01 last 106.95) total: 4500 /unlink in 36.97 seconds: 121.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 874 (time 1784052137.44 total 10.01 last 87.33) - open/close 1447 (time 1784052147.44 total 20.02 last 57.26) - open/close 2162 (time 1784052157.45 total 30.02 last 71.47) - open/close 2911 (time 1784052167.45 total 40.03 last 74.84) - open/close 3677 (time 1784052177.47 total 50.04 last 76.50) - open/close 4330 (time 1784052187.47 total 60.05 last 65.26) total: 4500 open/close in 62.81 seconds: 71.65 ops/second - /unlink 1509 (time 1784052204.75 total 10.00 last 150.87) - /unlink 2569 (time 1784052214.75 total 20.00 last 105.98) - /unlink 3599 (time 1784052224.76 total 30.01 last 102.95) total: 4500 /unlink in 39.62 seconds: 113.58 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11243: 95456 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg129-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg129-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg129-server stop mds service on oleg129-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg129-server stop mds service on oleg129-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg129-server unloading modules via unload_modules_local on: 'oleg129-server' oleg129-server: oleg129-server.virtnet: executing unload_modules_local oleg129-server: modules unloaded.