== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 19:55:45 (1787615745) Stopping clients: oleg628-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg628-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg628-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg628-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg628-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg628-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg628-server oleg628-server: oleg628-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg628-server' oleg628-server: oleg628-server.virtnet: executing load_modules_local oleg628-server: Loading modules from /home/green/git/lustre-release/lustre oleg628-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg628-server: detected 4 online CPUs by sysfs oleg628-server: MODOPTS_LIBCFS= oleg628-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg628-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg628-server: oleg628-server.virtnet: executing set_default_debug -1 all pdsh@oleg628-client: oleg628-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg628-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg628-server: oleg628-server.virtnet: executing set_default_debug -1 all pdsh@oleg628-client: oleg628-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg628-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg628-server: oleg628-server.virtnet: executing set_default_debug -1 all pdsh@oleg628-client: oleg628-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg628-client.virtnet: -o user_xattr,flock 192.168.206.128@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 746 (time 1787615953.90 total 10.00 last 74.58) - open/close 1445 (time 1787615963.91 total 20.01 last 69.82) - open/close 2150 (time 1787615973.93 total 30.04 last 70.34) - open/close 2891 (time 1787615983.94 total 40.04 last 74.07) - open/close 3624 (time 1787615993.94 total 50.04 last 73.27) - open/close 4308 (time 1787616003.97 total 60.07 last 68.23) total: 4500 open/close in 62.77 seconds: 71.69 ops/second - /unlink 1218 (time 1787616022.39 total 10.02 last 121.58) - /unlink 2565 (time 1787616032.39 total 20.02 last 134.69) - /unlink 3943 (time 1787616042.39 total 30.02 last 137.77) total: 4500 /unlink in 34.08 seconds: 132.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 740 (time 1787616061.29 total 10.01 last 73.93) - open/close 1459 (time 1787616071.29 total 20.01 last 71.87) - open/close 2210 (time 1787616081.30 total 30.02 last 75.03) - open/close 2958 (time 1787616091.31 total 40.03 last 74.74) - open/close 3703 (time 1787616101.31 total 50.03 last 74.47) - open/close 4422 (time 1787616111.33 total 60.05 last 71.79) total: 4500 open/close in 61.10 seconds: 73.65 ops/second - /unlink 1458 (time 1787616126.65 total 10.00 last 145.75) - /unlink 2836 (time 1787616136.65 total 20.01 last 137.73) - /unlink 4224 (time 1787616146.66 total 30.01 last 138.74) total: 4500 /unlink in 31.89 seconds: 141.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 721 (time 1787616162.34 total 10.01 last 72.06) - open/close 1441 (time 1787616172.35 total 20.01 last 71.95) - open/close 2180 (time 1787616182.36 total 30.02 last 73.82) - open/close 2930 (time 1787616192.37 total 40.03 last 74.98) - open/close 3643 (time 1787616202.37 total 50.03 last 71.28) - open/close 4408 (time 1787616212.37 total 60.03 last 76.48) total: 4500 open/close in 61.31 seconds: 73.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1292 (time 1787616227.19 total 10.00 last 129.18) - /unlink 2635 (time 1787616237.19 total 20.01 last 134.24) - /unlink 3833 (time 1787616247.20 total 30.01 last 119.74) total: 4500 /unlink in 35.39 seconds: 127.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 710 (time 1787616267.02 total 10.01 last 70.94) - open/close 1405 (time 1787616277.02 total 20.02 last 69.46) - open/close 2130 (time 1787616287.04 total 30.03 last 72.42) - open/close 2844 (time 1787616297.04 total 40.03 last 71.39) - open/close 3604 (time 1787616307.05 total 50.04 last 75.90) - open/close 4342 (time 1787616317.06 total 60.05 last 73.75) total: 4500 open/close in 62.15 seconds: 72.40 ops/second - /unlink 1451 (time 1787616333.87 total 10.00 last 145.08) - /unlink 2791 (time 1787616343.88 total 20.00 last 133.97) - /unlink 4104 (time 1787616353.88 total 30.01 last 131.26) total: 4500 /unlink in 33.15 seconds: 135.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 594 (time 1787616372.31 total 10.03 last 59.22) - open/close 1142 (time 1787616382.32 total 20.04 last 54.74) - open/close 1876 (time 1787616392.33 total 30.05 last 73.35) - open/close 2622 (time 1787616402.34 total 40.06 last 74.55) - open/close 3393 (time 1787616412.36 total 50.08 last 76.90) - open/close 4114 (time 1787616422.36 total 60.08 last 72.09) total: 4500 open/close in 65.81 seconds: 68.38 ops/second - /unlink 1361 (time 1787616443.02 total 10.00 last 136.05) - /unlink 2676 (time 1787616453.02 total 20.01 last 131.47) - /unlink 3972 (time 1787616463.02 total 30.01 last 129.58) total: 4500 /unlink in 34.32 seconds: 131.11 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 643 (time 1787616482.18 total 10.01 last 64.25) - open/close 1382 (time 1787616492.19 total 20.02 last 73.79) - open/close 2005 (time 1787616502.20 total 30.03 last 62.25) - open/close 2685 (time 1787616512.20 total 40.03 last 67.99) - open/close 3341 (time 1787616522.22 total 50.05 last 65.51) - open/close 4004 (time 1787616532.23 total 60.06 last 66.23) - open/close 4394 (time 1787616542.24 total 70.07 last 38.97) total: 4500 open/close in 73.02 seconds: 61.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 852 (time 1787616564.06 total 10.01 last 85.13) - /unlink 2031 (time 1787616574.07 total 20.01 last 117.87) - /unlink 3309 (time 1787616584.07 total 30.02 last 127.71) total: 4500 /unlink in 39.76 seconds: 113.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 703 (time 1787616609.30 total 10.01 last 70.22) - open/close 1332 (time 1787616619.31 total 20.01 last 62.88) - open/close 2051 (time 1787616629.31 total 30.02 last 71.86) - open/close 2794 (time 1787616639.32 total 40.03 last 74.22) - open/close 3582 (time 1787616649.34 total 50.04 last 78.71) - open/close 4282 (time 1787616659.34 total 60.05 last 69.98) total: 4500 open/close in 63.04 seconds: 71.38 ops/second - /unlink 1265 (time 1787616677.67 total 10.00 last 126.48) - /unlink 2406 (time 1787616687.67 total 20.01 last 114.05) - /unlink 3741 (time 1787616697.68 total 30.01 last 133.43) total: 4500 /unlink in 35.94 seconds: 125.21 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 665 (time 1787616718.61 total 10.01 last 66.43) - open/close 1380 (time 1787616728.62 total 20.02 last 71.44) - open/close 2083 (time 1787616738.62 total 30.02 last 70.28) - open/close 2759 (time 1787616748.63 total 40.03 last 67.57) - open/close 3524 (time 1787616758.63 total 50.03 last 76.45) - open/close 4171 (time 1787616768.65 total 60.05 last 64.60) total: 4500 open/close in 66.62 seconds: 67.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1193 (time 1787616790.51 total 10.00 last 119.27) - /unlink 2500 (time 1787616800.52 total 20.01 last 130.61) - /unlink 3936 (time 1787616810.52 total 30.01 last 143.57) total: 4500 /unlink in 34.44 seconds: 130.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 626 (time 1787616830.04 total 10.01 last 62.56) - open/close 1380 (time 1787616840.04 total 20.01 last 75.35) - open/close 2064 (time 1787616850.04 total 30.01 last 68.40) - open/close 2729 (time 1787616860.05 total 40.02 last 66.46) - open/close 3436 (time 1787616870.06 total 50.03 last 70.66) - open/close 4084 (time 1787616880.06 total 60.03 last 64.77) total: 4500 open/close in 66.10 seconds: 68.08 ops/second - /unlink 1355 (time 1787616900.33 total 10.01 last 135.40) - /unlink 2604 (time 1787616910.33 total 20.01 last 124.89) - /unlink 3853 (time 1787616920.33 total 30.01 last 124.89) total: 4500 /unlink in 34.61 seconds: 130.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 632 (time 1787616941.39 total 10.01 last 63.15) - open/close 1294 (time 1787616951.40 total 20.02 last 66.12) - open/close 1954 (time 1787616961.42 total 30.03 last 65.92) - open/close 2578 (time 1787616971.42 total 40.03 last 62.39) - open/close 3176 (time 1787616981.43 total 50.05 last 59.70) - open/close 3827 (time 1787616991.44 total 60.06 last 65.03) total: 4500 open/close in 68.92 seconds: 65.29 ops/second - /unlink 1417 (time 1787617014.35 total 10.00 last 141.69) - /unlink 2850 (time 1787617024.35 total 20.01 last 143.22) - /unlink 4116 (time 1787617034.35 total 30.01 last 126.57) total: 4500 /unlink in 32.66 seconds: 137.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 669 (time 1787617051.61 total 10.01 last 66.85) - open/close 1372 (time 1787617061.61 total 20.01 last 70.30) - open/close 2005 (time 1787617071.61 total 30.01 last 63.27) - open/close 2679 (time 1787617081.62 total 40.02 last 67.37) - open/close 3337 (time 1787617091.64 total 50.04 last 65.66) - open/close 3985 (time 1787617101.65 total 60.05 last 64.74) total: 4500 open/close in 67.29 seconds: 66.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1348 (time 1787617123.07 total 10.00 last 134.77) - /unlink 2759 (time 1787617133.08 total 20.01 last 141.01) - /unlink 3998 (time 1787617143.08 total 30.01 last 123.90) total: 4500 /unlink in 33.77 seconds: 133.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 603 (time 1787617162.22 total 10.00 last 60.28) - open/close 1158 (time 1787617172.22 total 20.01 last 55.48) - open/close 1915 (time 1787617182.23 total 30.02 last 75.62) - open/close 2634 (time 1787617192.25 total 40.03 last 71.78) - open/close 3408 (time 1787617202.27 total 50.06 last 77.23) - open/close 4106 (time 1787617212.27 total 60.06 last 69.78) total: 4500 open/close in 65.90 seconds: 68.29 ops/second - /unlink 1181 (time 1787617232.80 total 10.01 last 118.01) - /unlink 2405 (time 1787617242.80 total 20.01 last 122.37) - /unlink 3526 (time 1787617252.80 total 30.01 last 112.07) total: 4500 /unlink in 39.37 seconds: 114.31 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 698 (time 1787617277.04 total 10.00 last 69.77) - open/close 1312 (time 1787617287.07 total 20.04 last 61.21) - open/close 1936 (time 1787617297.07 total 30.04 last 62.40) - open/close 2726 (time 1787617307.08 total 40.05 last 78.92) - open/close 3488 (time 1787617317.09 total 50.05 last 76.16) - open/close 4261 (time 1787617327.10 total 60.06 last 77.23) total: 4500 open/close in 63.24 seconds: 71.16 ops/second - /unlink 1350 (time 1787617344.97 total 10.01 last 134.88) - /unlink 2674 (time 1787617354.98 total 20.02 last 132.28) - /unlink 4209 (time 1787617364.98 total 30.02 last 153.46) total: 4500 /unlink in 33.22 seconds: 135.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 717 (time 1787617384.53 total 10.01 last 71.66) - open/close 1409 (time 1787617394.53 total 20.01 last 69.19) - open/close 1974 (time 1787617404.55 total 30.02 last 56.42) - open/close 2568 (time 1787617414.55 total 40.03 last 59.36) - open/close 3337 (time 1787617424.56 total 50.03 last 76.86) - open/close 4098 (time 1787617434.56 total 60.04 last 76.09) total: 4500 open/close in 65.43 seconds: 68.77 ops/second - /unlink 1345 (time 1787617454.17 total 10.01 last 134.36) - /unlink 2522 (time 1787617464.17 total 20.01 last 117.68) - /unlink 3709 (time 1787617474.18 total 30.02 last 118.63) total: 4500 /unlink in 37.16 seconds: 121.11 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 647 (time 1787617496.99 total 10.03 last 64.48) - open/close 1251 (time 1787617507.00 total 20.05 last 60.31) - open/close 1858 (time 1787617517.02 total 30.06 last 60.61) - open/close 2512 (time 1787617527.04 total 40.09 last 65.24) - open/close 3243 (time 1787617537.05 total 50.09 last 73.04) - open/close 3998 (time 1787617547.07 total 60.12 last 75.33) total: 4500 open/close in 68.00 seconds: 66.17 ops/second - /unlink 1437 (time 1787617569.24 total 10.00 last 143.68) - /unlink 2640 (time 1787617579.24 total 20.00 last 120.27) - /unlink 3926 (time 1787617589.24 total 30.01 last 128.54) total: 4500 /unlink in 35.06 seconds: 128.36 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 621 (time 1787617609.06 total 10.00 last 62.08) - open/close 1278 (time 1787617619.07 total 20.02 last 65.61) - open/close 2012 (time 1787617629.07 total 30.02 last 73.39) - open/close 2764 (time 1787617639.10 total 40.04 last 75.01) - open/close 3449 (time 1787617649.12 total 50.07 last 68.31) - open/close 4105 (time 1787617659.12 total 60.07 last 65.59) total: 4500 open/close in 65.47 seconds: 68.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1481 (time 1787617678.82 total 10.00 last 148.08) - /unlink 2875 (time 1787617688.82 total 20.01 last 139.32) - /unlink 4208 (time 1787617698.83 total 30.01 last 133.21) total: 4500 /unlink in 33.03 seconds: 136.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 756 (time 1787617718.01 total 10.00 last 75.57) - open/close 1501 (time 1787617728.02 total 20.01 last 74.45) - open/close 2283 (time 1787617738.02 total 30.02 last 78.15) - open/close 3102 (time 1787617748.03 total 40.02 last 81.84) - open/close 3900 (time 1787617758.05 total 50.04 last 79.65) total: 4500 open/close in 57.27 seconds: 78.57 ops/second - /unlink 1429 (time 1787617778.93 total 10.00 last 142.86) - /unlink 2863 (time 1787617788.94 total 20.01 last 143.32) - /unlink 4181 (time 1787617798.94 total 30.01 last 131.80) total: 4500 /unlink in 32.18 seconds: 139.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 680 (time 1787617815.90 total 10.01 last 67.96) - open/close 1329 (time 1787617825.90 total 20.01 last 64.89) - open/close 2014 (time 1787617835.90 total 30.01 last 68.48) - open/close 2746 (time 1787617845.91 total 40.02 last 73.11) - open/close 3453 (time 1787617855.92 total 50.03 last 70.63) - open/close 4153 (time 1787617865.93 total 60.04 last 69.93) total: 4500 open/close in 64.42 seconds: 69.85 ops/second - /unlink 1402 (time 1787617883.86 total 10.00 last 140.18) - /unlink 2811 (time 1787617893.86 total 20.00 last 140.89) - /unlink 4157 (time 1787617903.86 total 30.00 last 134.57) total: 4500 /unlink in 32.43 seconds: 138.77 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95227 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg628-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg628-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg628-server stop mds service on oleg628-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg628-server stop mds service on oleg628-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg628-server unloading modules via unload_modules_local on: 'oleg628-server' oleg628-server: oleg628-server.virtnet: executing unload_modules_local oleg628-server: modules unloaded.