== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 12:26:46 (1789144006) Stopping clients: oleg414-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg414-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg414-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg414-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg414-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg414-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg414-server oleg414-server: oleg414-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg414-server' oleg414-server: oleg414-server.virtnet: executing load_modules_local oleg414-server: Loading modules from /home/green/git/lustre-release/lustre oleg414-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg414-server: detected 4 online CPUs by sysfs oleg414-server: MODOPTS_LIBCFS= oleg414-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg414-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg414-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg414-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg414-client.virtnet: -o user_xattr,flock 192.168.204.114@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 681 (time 1789144212.05 total 10.01 last 68.05) - open/close 1285 (time 1789144222.05 total 20.01 last 60.39) - open/close 1761 (time 1789144232.05 total 30.01 last 47.59) - open/close 2410 (time 1789144242.08 total 40.04 last 64.72) - open/close 3010 (time 1789144252.08 total 50.04 last 60.00) - open/close 3729 (time 1789144262.11 total 60.06 last 71.73) - open/close 4435 (time 1789144272.11 total 70.06 last 70.60) total: 4500 open/close in 71.16 seconds: 63.24 ops/second - /unlink 1370 (time 1789144288.92 total 10.00 last 136.98) - /unlink 2664 (time 1789144298.93 total 20.01 last 129.34) - /unlink 3909 (time 1789144308.94 total 30.02 last 124.34) total: 4500 /unlink in 34.94 seconds: 128.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 502 (time 1789144329.49 total 10.01 last 50.14) - open/close 972 (time 1789144339.50 total 20.02 last 46.94) - open/close 1586 (time 1789144349.50 total 30.03 last 61.38) - open/close 2259 (time 1789144359.50 total 40.03 last 67.30) - open/close 2895 (time 1789144369.51 total 50.03 last 63.57) - open/close 3535 (time 1789144379.51 total 60.04 last 63.97) - open/close 4168 (time 1789144389.52 total 70.04 last 63.25) total: 4500 open/close in 74.33 seconds: 60.54 ops/second - /unlink 1273 (time 1789144408.43 total 10.00 last 127.24) - /unlink 2488 (time 1789144418.43 total 20.01 last 121.48) - /unlink 3833 (time 1789144428.44 total 30.01 last 134.46) total: 4500 /unlink in 35.23 seconds: 127.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 675 (time 1789144449.12 total 10.00 last 67.50) - open/close 1353 (time 1789144459.12 total 20.00 last 67.80) - open/close 1914 (time 1789144469.12 total 30.01 last 56.05) - open/close 2543 (time 1789144479.13 total 40.01 last 62.89) - open/close 3207 (time 1789144489.14 total 50.02 last 66.34) - open/close 3837 (time 1789144499.14 total 60.02 last 63.00) - open/close 4464 (time 1789144509.14 total 70.03 last 62.66) total: 4500 open/close in 70.47 seconds: 63.85 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1215 (time 1789144524.92 total 10.00 last 121.48) - /unlink 2337 (time 1789144534.94 total 20.01 last 112.07) - /unlink 3543 (time 1789144544.94 total 30.02 last 120.51) total: 4500 /unlink in 37.12 seconds: 121.23 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 680 (time 1789144567.05 total 10.00 last 68.00) - open/close 1304 (time 1789144577.06 total 20.01 last 62.33) - open/close 1876 (time 1789144587.06 total 30.02 last 57.17) - open/close 2418 (time 1789144597.10 total 40.05 last 54.02) - open/close 3077 (time 1789144607.11 total 50.06 last 65.82) - open/close 3683 (time 1789144617.14 total 60.10 last 60.40) - open/close 4237 (time 1789144627.15 total 70.10 last 55.35) total: 4500 open/close in 74.74 seconds: 60.21 ops/second - /unlink 1264 (time 1789144647.62 total 10.01 last 126.28) - /unlink 2413 (time 1789144657.62 total 20.01 last 114.85) - /unlink 3733 (time 1789144667.63 total 30.02 last 131.96) total: 4500 /unlink in 36.96 seconds: 121.76 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 387 (time 1789144692.34 total 10.05 last 38.51) - open/close 855 (time 1789144702.35 total 20.06 last 46.74) - open/close 1348 (time 1789144712.35 total 30.07 last 49.28) - open/close 1902 (time 1789144722.36 total 40.08 last 55.35) - open/close 2647 (time 1789144732.37 total 50.08 last 74.45) - open/close 3378 (time 1789144742.41 total 60.12 last 72.80) - open/close 3914 (time 1789144752.42 total 70.14 last 53.54) - open/close 4449 (time 1789144762.44 total 80.15 last 53.41) total: 4500 open/close in 80.93 seconds: 55.60 ops/second - /unlink 1109 (time 1789144778.28 total 10.01 last 110.81) - /unlink 2396 (time 1789144788.28 total 20.01 last 128.63) - /unlink 3664 (time 1789144798.29 total 30.02 last 126.77) total: 4500 /unlink in 37.06 seconds: 121.42 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 599 (time 1789144820.89 total 10.00 last 59.90) - open/close 1257 (time 1789144830.91 total 20.02 last 65.69) - open/close 1887 (time 1789144840.91 total 30.02 last 63.00) - open/close 2581 (time 1789144850.92 total 40.02 last 69.34) - open/close 3272 (time 1789144860.92 total 50.03 last 69.03) - open/close 4001 (time 1789144870.93 total 60.04 last 72.85) total: 4500 open/close in 67.63 seconds: 66.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1071 (time 1789144894.21 total 10.01 last 107.02) - /unlink 2214 (time 1789144904.21 total 20.01 last 114.29) - /unlink 3378 (time 1789144914.23 total 30.03 last 116.15) - /unlink 4450 (time 1789144924.23 total 40.03 last 107.18) total: 4500 /unlink in 40.84 seconds: 110.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 545 (time 1789144941.60 total 10.01 last 54.46) - open/close 1088 (time 1789144951.62 total 20.02 last 54.23) - open/close 1635 (time 1789144961.65 total 30.05 last 54.54) - open/close 2324 (time 1789144971.66 total 40.06 last 68.80) - open/close 2975 (time 1789144981.67 total 50.07 last 65.06) - open/close 3520 (time 1789144991.68 total 60.08 last 54.46) - open/close 4163 (time 1789145001.69 total 70.09 last 64.21) total: 4500 open/close in 75.75 seconds: 59.40 ops/second - /unlink 1262 (time 1789145022.93 total 10.01 last 126.08) - /unlink 2507 (time 1789145032.94 total 20.02 last 124.42) - /unlink 3755 (time 1789145042.95 total 30.02 last 124.69) total: 4500 /unlink in 36.36 seconds: 123.75 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 635 (time 1789145064.53 total 10.00 last 63.48) - open/close 1351 (time 1789145074.55 total 20.02 last 71.44) - open/close 1887 (time 1789145084.57 total 30.04 last 53.51) - open/close 2392 (time 1789145094.59 total 40.06 last 50.40) - open/close 2890 (time 1789145104.61 total 50.09 last 49.68) - open/close 3384 (time 1789145114.62 total 60.09 last 49.40) - open/close 3964 (time 1789145124.62 total 70.09 last 57.98) - open/close 4471 (time 1789145134.64 total 80.11 last 50.59) total: 4500 open/close in 80.48 seconds: 55.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1151 (time 1789145150.06 total 10.01 last 114.95) - /unlink 2440 (time 1789145160.07 total 20.02 last 128.83) - /unlink 3657 (time 1789145170.07 total 30.02 last 121.65) total: 4500 /unlink in 37.80 seconds: 119.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 489 (time 1789145195.03 total 10.03 last 48.73) - open/close 1087 (time 1789145205.04 total 20.04 last 59.74) - open/close 1733 (time 1789145215.07 total 30.06 last 64.47) - open/close 2403 (time 1789145225.08 total 40.08 last 66.90) - open/close 3104 (time 1789145235.11 total 50.11 last 69.90) - open/close 3701 (time 1789145245.12 total 60.12 last 59.66) - open/close 4302 (time 1789145255.14 total 70.13 last 59.98) total: 4500 open/close in 74.22 seconds: 60.63 ops/second - /unlink 1207 (time 1789145274.36 total 10.00 last 120.66) - /unlink 2394 (time 1789145284.37 total 20.01 last 118.60) - /unlink 3670 (time 1789145294.39 total 30.03 last 127.36) total: 4500 /unlink in 36.37 seconds: 123.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 658 (time 1789145316.34 total 10.01 last 65.76) - open/close 1282 (time 1789145326.34 total 20.01 last 62.37) - open/close 1845 (time 1789145336.35 total 30.02 last 56.26) - open/close 2341 (time 1789145346.37 total 40.03 last 49.53) - open/close 2814 (time 1789145356.37 total 50.03 last 47.30) - open/close 3400 (time 1789145366.37 total 60.03 last 58.59) - open/close 3999 (time 1789145376.39 total 70.05 last 59.79) total: 4500 open/close in 77.80 seconds: 57.84 ops/second - /unlink 1315 (time 1789145399.25 total 10.00 last 131.45) - /unlink 2501 (time 1789145409.26 total 20.01 last 118.56) - /unlink 3717 (time 1789145419.26 total 30.01 last 121.54) total: 4500 /unlink in 36.49 seconds: 123.31 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 522 (time 1789145441.36 total 10.01 last 52.16) - open/close 1073 (time 1789145451.40 total 20.05 last 54.89) - open/close 1739 (time 1789145461.41 total 30.06 last 66.53) - open/close 2447 (time 1789145471.43 total 40.07 last 70.70) - open/close 3213 (time 1789145481.44 total 50.08 last 76.52) - open/close 3870 (time 1789145491.44 total 60.09 last 65.66) total: 4500 open/close in 68.41 seconds: 65.78 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1185 (time 1789145514.84 total 10.01 last 118.41) - /unlink 2190 (time 1789145524.85 total 20.01 last 100.47) - /unlink 3047 (time 1789145534.87 total 30.04 last 85.48) - /unlink 4022 (time 1789145544.87 total 40.04 last 97.48) total: 4500 /unlink in 45.31 seconds: 99.31 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 613 (time 1789145566.66 total 10.01 last 61.23) - open/close 1244 (time 1789145576.68 total 20.02 last 63.02) - open/close 1828 (time 1789145586.68 total 30.03 last 58.35) - open/close 2447 (time 1789145596.70 total 40.05 last 61.81) - open/close 3037 (time 1789145606.71 total 50.06 last 58.95) - open/close 3566 (time 1789145616.71 total 60.06 last 52.89) - open/close 4014 (time 1789145626.72 total 70.07 last 44.76) total: 4500 open/close in 78.51 seconds: 57.32 ops/second - /unlink 1231 (time 1789145652.09 total 10.00 last 123.07) - /unlink 2361 (time 1789145662.10 total 20.01 last 112.89) - /unlink 3521 (time 1789145672.10 total 30.01 last 115.99) total: 4500 /unlink in 38.24 seconds: 117.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 652 (time 1789145695.63 total 10.01 last 65.15) - open/close 1313 (time 1789145705.65 total 20.03 last 65.97) - open/close 1997 (time 1789145715.68 total 30.05 last 68.22) - open/close 2670 (time 1789145725.68 total 40.06 last 67.24) - open/close 3315 (time 1789145735.68 total 50.06 last 64.50) - open/close 3976 (time 1789145745.69 total 60.07 last 66.07) total: 4500 open/close in 67.27 seconds: 66.89 ops/second - /unlink 1096 (time 1789145767.88 total 10.00 last 109.59) - /unlink 2378 (time 1789145777.88 total 20.00 last 128.16) - /unlink 3662 (time 1789145787.88 total 30.01 last 128.34) total: 4500 /unlink in 36.31 seconds: 123.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 582 (time 1789145810.05 total 10.04 last 57.98) - open/close 1214 (time 1789145820.05 total 20.05 last 63.16) - open/close 1728 (time 1789145830.08 total 30.07 last 51.27) - open/close 2316 (time 1789145840.08 total 40.07 last 58.78) - open/close 2814 (time 1789145850.11 total 50.10 last 49.66) - open/close 3430 (time 1789145860.12 total 60.11 last 61.55) - open/close 3981 (time 1789145870.13 total 70.12 last 55.03) total: 4500 open/close in 77.38 seconds: 58.16 ops/second - /unlink 1208 (time 1789145893.16 total 10.00 last 120.76) - /unlink 2404 (time 1789145903.16 total 20.01 last 119.54) - /unlink 3639 (time 1789145913.17 total 30.01 last 123.46) total: 4500 /unlink in 37.46 seconds: 120.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 611 (time 1789145937.41 total 10.00 last 61.09) - open/close 1139 (time 1789145947.42 total 20.01 last 52.75) - open/close 1731 (time 1789145957.42 total 30.02 last 59.16) - open/close 2354 (time 1789145967.43 total 40.02 last 62.27) - open/close 2961 (time 1789145977.43 total 50.03 last 60.65) - open/close 3600 (time 1789145987.44 total 60.03 last 63.87) - open/close 4272 (time 1789145997.45 total 70.04 last 67.12) total: 4500 open/close in 73.46 seconds: 61.25 ops/second - /unlink 1244 (time 1789146016.19 total 10.01 last 124.23) - /unlink 2324 (time 1789146026.19 total 20.02 last 107.96) - /unlink 3467 (time 1789146036.19 total 30.02 last 114.23) - /unlink 4498 (time 1789146046.20 total 40.03 last 103.02) total: 4500 /unlink in 40.04 seconds: 112.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 486 (time 1789146062.70 total 10.03 last 48.43) - open/close 954 (time 1789146072.73 total 20.07 last 46.64) - open/close 1404 (time 1789146082.74 total 30.08 last 44.97) - open/close 1841 (time 1789146092.75 total 40.09 last 43.66) - open/close 2439 (time 1789146102.76 total 50.10 last 59.74) - open/close 2995 (time 1789146112.76 total 60.10 last 55.56) - open/close 3466 (time 1789146122.78 total 70.12 last 47.04) - open/close 4016 (time 1789146132.78 total 80.12 last 54.96) total: 4500 open/close in 89.42 seconds: 50.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1142 (time 1789146157.36 total 10.00 last 114.19) - /unlink 2204 (time 1789146167.36 total 20.00 last 106.19) - /unlink 3321 (time 1789146177.36 total 30.00 last 111.67) total: 4500 /unlink in 39.55 seconds: 113.78 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 649 (time 1789146202.06 total 10.01 last 64.84) - open/close 1266 (time 1789146212.06 total 20.01 last 61.67) - open/close 1895 (time 1789146222.11 total 30.06 last 62.59) - open/close 2511 (time 1789146232.12 total 40.08 last 61.53) - open/close 3055 (time 1789146242.15 total 50.10 last 54.28) - open/close 3663 (time 1789146252.15 total 60.11 last 60.75) - open/close 4354 (time 1789146262.20 total 70.15 last 68.78) total: 4500 open/close in 72.44 seconds: 62.12 ops/second - /unlink 1244 (time 1789146280.05 total 10.00 last 124.36) - /unlink 2464 (time 1789146290.06 total 20.01 last 121.87) - /unlink 3678 (time 1789146300.06 total 30.02 last 121.34) total: 4500 /unlink in 37.15 seconds: 121.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 650 (time 1789146322.50 total 10.00 last 64.99) - open/close 1270 (time 1789146332.50 total 20.00 last 62.00) - open/close 1890 (time 1789146342.51 total 30.01 last 61.94) - open/close 2444 (time 1789146352.52 total 40.01 last 55.39) - open/close 3084 (time 1789146362.52 total 50.02 last 63.98) - open/close 3708 (time 1789146372.52 total 60.02 last 62.38) - open/close 4354 (time 1789146382.54 total 70.04 last 64.48) total: 4500 open/close in 72.12 seconds: 62.39 ops/second - /unlink 1374 (time 1789146399.75 total 10.00 last 137.40) - /unlink 2528 (time 1789146409.76 total 20.01 last 115.32) - /unlink 3660 (time 1789146419.76 total 30.01 last 113.18) total: 4500 /unlink in 37.16 seconds: 121.09 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11449: 95335 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg414-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg414-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg414-server stop mds service on oleg414-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg414-server stop mds service on oleg414-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg414-server unloading modules via unload_modules_local on: 'oleg414-server' oleg414-server: oleg414-server.virtnet: executing unload_modules_local oleg414-server: modules unloaded.