Random sampling#

Random sampling is often used in machine learning data-wrangling pipelines when choosing which data to train and which data to test against. Sampling a random fraction of a database table of 1 million records can be very useful (computationally effective) for practicing other methods on them and testing out new ideas.

Importing libraries and packages#

1# Mathematical operations and data manipulation
2import pandas as pd

Set paths#

1# Path to datasets directory
2data_path = "./datasets"
3# Path to assets directory (for saving results to)
4assets_path = "./assets"

Loading dataset#

1dataset = pd.read_csv(f"{data_path}/cleaned_mpi_disagg_by_groups.csv")

Wrangling#

1dataset.head()
Country Type of survey Survey year Ethnic/racial/caste group MPI: Value for the country MPI: Value for the group Headcount (%) Number of multidimensionally poor people by group (thousands) Intensity of deprivation (%) Health (%) ... Cooking fuel (%) Sanitation (%) Drinking water (%) Electricity (%) Housing (%) Assets (%) Population share by group (%) Population size by group (thousands) Population size (thousands) Region
0 Bangladesh MICS 2019 Bengali 0.104060 0.102702 24.384759 39284.990511 42.117223 17.441109 ... 12.484664 8.274627 0.569494 2.308714 12.478603 8.562996 98.809242 161104.688057 163046.173 South Asia
1 Bangladesh MICS 2019 Other 0.104060 0.216783 45.868093 890.521140 47.262356 10.881517 ... 11.733451 10.198139 8.354676 8.593331 11.536150 10.738271 1.190756 1941.482818 163046.173 South Asia
2 Belize MICS 2015/2016 Creole 0.017109 0.003768 1.051818 0.940881 35.820526 52.086931 ... 1.126231 3.964365 1.126231 3.383591 6.162911 4.409921 22.916001 89.452839 390.351 Latin America and the Caribbean
3 Belize MICS 2015/2016 Garifuna 0.017109 0.003887 1.097083 0.224891 35.433114 85.184902 ... 2.963020 2.963020 0.000000 2.963020 2.963020 2.963020 5.251431 20.499014 390.351 Latin America and the Caribbean
4 Belize MICS 2015/2016 Maya 0.017109 0.078922 18.631953 8.557940 42.358151 37.911840 ... 11.931632 7.811719 2.319572 9.465594 11.165109 4.267081 11.766724 45.931523 390.351 Latin America and the Caribbean

5 rows × 26 columns

1# Specify the number of samples with n
2dataset.sample(n=5)
Country Type of survey Survey year Ethnic/racial/caste group MPI: Value for the country MPI: Value for the group Headcount (%) Number of multidimensionally poor people by group (thousands) Intensity of deprivation (%) Health (%) ... Cooking fuel (%) Sanitation (%) Drinking water (%) Electricity (%) Housing (%) Assets (%) Population share by group (%) Population size by group (thousands) Population size (thousands) Region
120 Guinea-Bissau MICS 2019 Fula 0.340689 0.411074 74.504369 463.490298 55.174446 18.493306 ... 10.023939 9.706169 5.215410 6.833582 9.903508 1.121102 32.385477 622.098141 1920.917 Sub-Saharan Africa
43 Chad MICS 2019 Karo/Zimé 0.517011 0.384718 72.419661 85.829460 53.123480 29.356452 ... 10.370462 9.661809 5.951971 10.081941 10.401402 4.468767 0.743197 118.516793 15946.882 Sub-Saharan Africa
290 Uganda DHS 2016 Iteso 0.281028 0.308510 63.502318 2146.800347 48.582491 25.717343 ... 11.430293 8.974657 8.767943 11.207040 10.066991 4.533935 7.636540 3380.664559 44269.587 Sub-Saharan Africa
140 Kenya DHS 2014 Kalenjin 0.170776 0.206236 46.477121 2786.279675 44.373757 29.073624 ... 12.513454 10.200653 9.786458 12.267992 12.517002 6.527013 11.402885 5994.948929 52573.967 Sub-Saharan Africa
109 Guatemala DHS 2014/2015 Xinca 0.133518 0.203171 43.940374 54.206673 46.237841 28.915776 ... 11.873613 6.321930 4.400281 4.289283 10.693456 6.810853 0.701671 123.364159 17581.476 Latin America and the Caribbean

5 rows × 26 columns

1# Specify a definite fraction (percentage) of the data to be sampled
2dataset.sample(frac=0.1)
Country Type of survey Survey year Ethnic/racial/caste group MPI: Value for the country MPI: Value for the group Headcount (%) Number of multidimensionally poor people by group (thousands) Intensity of deprivation (%) Health (%) ... Cooking fuel (%) Sanitation (%) Drinking water (%) Electricity (%) Housing (%) Assets (%) Population share by group (%) Population size by group (thousands) Population size (thousands) Region
237 Senegal DHS 2019 Soninké 0.262862 0.130870 25.606647 87.993949 51.107776 26.923272 ... 10.870275 8.481950 3.704166 3.791185 5.204666 1.051831 2.108674 343.637138 16296.362 Sub-Saharan Africa
105 Guatemala DHS 2014/2015 Garifuna 0.133518 0.034574 8.890413 0.955083 38.888890 42.857143 ... 14.285715 0.000000 0.000000 0.000000 0.000000 0.000000 0.061103 10.742838 17581.476 Latin America and the Caribbean
130 Guyana MICS 2019/2020 Other ethnicity/DK/Missing 0.006592 0.000000 0.000000 0.000000 44.117537 0.000000 ... 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.198953 1.557356 782.775 Latin America and the Caribbean
43 Chad MICS 2019 Karo/Zimé 0.517011 0.384718 72.419661 85.829460 53.123480 29.356452 ... 10.370462 9.661809 5.951971 10.081941 10.401402 4.468767 0.743197 118.516793 15946.882 Sub-Saharan Africa
167 Malawi DHS 2015/2016 Nkhonde 0.252325 0.162463 40.232983 64.828205 40.380523 30.813015 ... 13.758008 4.926294 8.182173 13.758008 11.682096 6.829682 0.864964 161.131986 18628.749 Sub-Saharan Africa
7 Bolivia, Plurinational State of EDSA 2016 Aymara 0.037754 0.038037 10.026737 223.806399 37.935901 16.573222 ... 10.967489 14.212216 5.715832 2.869013 12.384772 5.318492 19.387443 2232.096130 11513.102 Latin America and the Caribbean
293 Uganda DHS 2016 Other 0.281028 0.348234 66.484243 5747.428972 52.378464 24.311363 ... 10.578003 9.857720 8.236721 9.430651 9.725867 5.491462 19.527625 8644.798738 44269.587 Sub-Saharan Africa
177 Mali DHS 2018 Malinke 0.376063 0.317270 59.140390 947.349021 53.646982 23.213191 ... 10.308281 7.143877 3.695189 6.452078 8.969188 1.202482 8.148656 1601.864679 19658.023 Sub-Saharan Africa
235 Senegal DHS 2019 Poular 0.262862 0.332379 58.777547 2617.060760 56.548661 19.393708 ... 9.361020 7.132895 4.602252 7.256661 6.652603 1.553368 27.321950 4452.483812 16296.362 Sub-Saharan Africa
57 Colombia DHS 2015/2016 Indigenous 0.019657 0.084061 19.114792 603.780403 43.976903 7.732241 ... 11.675702 10.996754 10.044707 6.387769 11.618928 5.119122 6.274816 3158.707622 50339.443 Latin America and the Caribbean
99 Ghana MICS 2017/2018 Gruma 0.111218 0.276510 54.131573 655.914777 51.081169 21.250694 ... 10.874234 9.649725 8.757892 5.434210 6.146713 2.421232 3.983530 1211.704621 30417.858 Sub-Saharan Africa
77 Gabon DHS 2012 Fang 0.069695 0.041707 10.098383 51.428649 41.300541 45.500579 ... 6.838636 11.894406 8.369483 6.335231 5.837094 3.215685 23.441096 509.276087 2172.578 Sub-Saharan Africa
66 Cote d'Ivoire MICS 2016 Other Ivorian 0.235871 0.241984 46.122292 86.125949 52.465838 22.833949 ... 9.904710 9.790189 4.386417 3.421665 4.162283 0.948647 0.726123 186.733889 25716.554 Sub-Saharan Africa
111 Guinea DHS 2018 Kissi 0.373222 0.407443 77.932626 93.539823 52.281505 21.437003 ... 10.496256 9.322660 6.313832 10.129435 7.585464 4.632572 5.524613 120.026526 2172.578 Sub-Saharan Africa
212 Paraguay MICS 2016 Spanish speaker 0.018849 0.001884 0.458962 6.327117 41.046467 15.870105 ... 12.250035 12.250035 6.775407 6.775407 5.474627 0.000000 19.569077 1378.570802 7044.639 Latin America and the Caribbean
151 Kenya DHS 2014 Somali 0.170776 0.431177 75.299144 1170.103478 57.261908 14.258599 ... 9.694605 8.319060 6.725044 8.635744 9.655286 6.462009 2.955721 1553.939951 52573.967 Sub-Saharan Africa
274 Togo MICS 2017 Other Togolese 0.179616 0.258282 51.249850 157.648656 50.396621 20.616874 ... 10.951374 10.477453 6.101929 9.072888 8.298033 3.666544 3.805919 307.608036 8082.359 Sub-Saharan Africa
69 Cuba MICS 2019 Mulato/Mestizo/Other 0.002689 0.003503 0.918265 23.120644 38.146138 9.268285 ... 9.529931 11.936319 4.327483 1.003049 14.563873 13.557607 22.216125 2517.860953 11333.484 Latin America and the Caribbean
195 Nigeria DHS 2018 Fulani 0.254390 0.561016 85.658127 9522.887814 65.494788 27.474762 ... 8.451475 6.223701 5.846892 6.905513 7.064566 3.207514 5.532007 11117.319675 200963.603 Sub-Saharan Africa
145 Kenya DHS 2014 Luo 0.170776 0.144795 34.381160 1845.365949 42.114729 31.043569 ... 12.992951 11.288903 9.570467 11.963744 13.117927 5.336890 10.209189 5367.375451 52573.967 Sub-Saharan Africa
249 Sierra Leone DHS 2019 Loko 0.292899 0.212481 47.473097 65.443718 44.758216 25.912872 ... 12.412370 10.870249 6.917696 9.775864 8.095606 6.234124 1.764376 137.854327 7813.207 Sub-Saharan Africa
29 Central African Republic MICS 2018/2019 Mboum 0.461348 0.544202 89.447999 362.646821 60.840046 20.541184 ... 9.068336 8.736534 6.783807 8.763939 8.618528 7.855962 8.543988 405.427539 4745.179 Sub-Saharan Africa
83 Gabon DHS 2012 Other 0.069695 0.111062 23.392190 43.069936 47.478026 32.666112 ... 7.110192 11.398305 8.269101 6.701042 7.847704 5.757293 8.474771 184.121009 2172.578 Sub-Saharan Africa
233 Senegal DHS 2019 Mandingue/ Socé 0.262862 0.279194 52.736539 420.763839 52.941322 28.339235 ... 10.157087 6.233958 5.502314 4.915290 6.794402 1.018511 4.895940 797.860175 16296.362 Sub-Saharan Africa
112 Guinea DHS 2018 Malinké 0.373222 0.384910 68.058491 433.423280 56.555790 23.422878 ... 9.791178 7.557901 5.115107 6.064878 5.972220 2.063464 29.312614 636.839394 2172.578 Sub-Saharan Africa
220 Peru ENDES 2018 White 0.029186 0.031832 8.089525 135.934503 39.349723 22.392238 ... 11.493846 11.854284 6.008284 4.364385 13.486895 4.266489 5.168727 1680.376919 32510.462 Latin America and the Caribbean
185 Moldova, Republic of MICS 2012 Gagauz 0.003534 0.000815 0.244517 0.373684 33.333334 0.000000 ... 0.000000 11.377531 16.666667 0.000000 5.289136 16.666667 3.779751 152.825072 4043.258 Europe and Central Asia
149 Kenya DHS 2014 Other 0.170776 0.266450 53.123373 1025.676048 50.156796 21.811049 ... 10.960653 10.047830 8.873462 10.482296 11.065171 6.771979 3.672433 1930.743465 52573.967 Sub-Saharan Africa
139 Kenya DHS 2014 Embu 0.170776 0.085377 21.533428 86.084593 39.648640 30.905186 ... 13.289951 12.917526 7.343862 11.644053 14.011970 5.640906 0.760399 399.771890 52573.967 Sub-Saharan Africa
24 Burkina Faso DHS 2010 Touareg/Bella 0.523424 0.700454 96.989411 382.144899 72.219616 21.819641 ... 7.663170 7.589635 5.620116 7.611773 7.560016 3.563987 1.938878 394.006824 20321.383 Sub-Saharan Africa

30 rows × 26 columns

1# With the default replace FALSE, there is no repetition, and sampling will
2# try to choose new elements only. With replace set to TRUE, the same record
3# can be chosen more than once.
4dataset.sample(frac=0.1, replace=True)
Country Type of survey Survey year Ethnic/racial/caste group MPI: Value for the country MPI: Value for the group Headcount (%) Number of multidimensionally poor people by group (thousands) Intensity of deprivation (%) Health (%) ... Cooking fuel (%) Sanitation (%) Drinking water (%) Electricity (%) Housing (%) Assets (%) Population share by group (%) Population size by group (thousands) Population size (thousands) Region
241 Serbia MICS 2019 Other/Does not want to declare 0.000433 0.000000 0.000000 0.000000 0.000000 0.000000 ... 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 5.105867 447.898315 8772.228 Europe and Central Asia
245 Sierra Leone DHS 2019 Fullah 0.292899 0.193856 41.186848 105.619194 47.067374 27.932823 ... 11.648995 9.939921 6.537680 9.082134 4.229454 4.752828 3.282124 256.439127 7813.207 Sub-Saharan Africa
191 Mongolia MICS 2018 Kazakh 0.028127 0.081602 20.678778 29.174153 39.461523 21.697732 ... 13.926272 13.993324 8.477549 1.266479 13.673975 1.001737 4.374428 141.082580 3225.166 East Asia and the Paficic
39 Chad MICS 2019 Dadjo/Mouro 0.517011 0.597212 90.557122 325.431899 65.948600 15.667545 ... 8.334058 7.848535 4.709626 8.188266 8.122917 6.125602 2.253522 359.366431 15946.882 Sub-Saharan Africa
279 Trinidad and Tobago MICS 2011 Other/Not stated 0.002418 0.000000 0.000000 0.000000 0.000000 0.000000 ... 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 1.655627 23.095487 1394.969 Latin America and the Caribbean
24 Burkina Faso DHS 2010 Touareg/Bella 0.523424 0.700454 96.989411 382.144899 72.219616 21.819641 ... 7.663170 7.589635 5.620116 7.611773 7.560016 3.563987 1.938878 394.006824 20321.383 Sub-Saharan Africa
66 Cote d'Ivoire MICS 2016 Other Ivorian 0.235871 0.241984 46.122292 86.125949 52.465838 22.833949 ... 9.904710 9.790189 4.386417 3.421665 4.162283 0.948647 0.726123 186.733889 25716.554 Sub-Saharan Africa
73 Ecuador ECV 2013/2014 Mestizo 0.018254 0.010980 2.840898 372.585856 38.649818 46.104187 ... 3.248481 7.520760 6.941742 1.503983 6.413955 3.650843 75.488281 13115.075060 17373.657 Latin America and the Caribbean
132 India DHS 2015/2016 Scheduled Caste 0.122652 0.146824 33.280160 94132.425201 44.117540 31.322577 ... 11.974340 11.428340 2.409617 4.115888 10.889990 4.286750 20.700000 282848.475492 1366417.756 South Asia
169 Malawi DHS 2015/2016 Other 0.252325 0.234464 48.556921 163.102389 48.286474 23.732468 ... 11.505407 5.905223 6.797042 11.429387 9.949313 6.214361 1.803124 335.899366 18628.749 Sub-Saharan Africa
58 Colombia DHS 2015/2016 None of the above 0.019657 0.013802 3.511228 1493.222257 39.308789 13.767309 ... 10.646211 9.107133 8.757170 3.172399 10.733576 2.339259 84.480613 42527.070264 50339.443 Latin America and the Caribbean
118 Guinea-Bissau MICS 2019 Beafada 0.340689 0.276899 58.936185 45.335174 46.982822 18.968948 ... 11.797836 11.215103 3.847717 9.098400 11.584077 1.513986 4.004467 76.922478 1920.917 Sub-Saharan Africa
120 Guinea-Bissau MICS 2019 Fula 0.340689 0.411074 74.504369 463.490298 55.174446 18.493306 ... 10.023939 9.706169 5.215410 6.833582 9.903508 1.121102 32.385477 622.098141 1920.917 Sub-Saharan Africa
117 Guinea-Bissau MICS 2019 Balanta 0.340689 0.330197 66.328031 248.996752 49.782467 20.768883 ... 11.129975 10.772929 7.343452 7.927965 11.060540 3.665601 19.542854 375.401996 1920.917 Sub-Saharan Africa
295 Viet Nam MICS 2013/2014 Kinh/Hoa 0.019334 0.011037 2.988247 2480.421534 36.934501 16.320269 ... 12.656878 11.753646 3.399943 0.674117 9.583682 2.956655 86.050278 83005.912049 96462.108 East Asia and the Paficic
53 Chad MICS 2019 Toupouri/Kera 0.517011 0.445451 77.916104 151.913283 57.170624 26.194139 ... 9.679107 9.355118 7.506499 9.471221 9.717501 4.332808 1.222624 194.970329 15946.882 Sub-Saharan Africa
101 Ghana MICS 2017/2018 Guan 0.111218 0.199176 41.347292 531.573413 48.171496 26.884381 ... 11.532868 11.434940 7.388756 3.660692 5.875351 2.404575 4.226565 1285.630536 30417.858 Sub-Saharan Africa
69 Cuba MICS 2019 Mulato/Mestizo/Other 0.002689 0.003503 0.918265 23.120644 38.146138 9.268285 ... 9.529931 11.936319 4.327483 1.003049 14.563873 13.557607 22.216125 2517.860953 11333.484 Latin America and the Caribbean
160 Lao People's Democratic Republic MICS 2017 Lao-Tai 0.108333 0.048035 10.904098 486.855744 44.052187 22.464549 ... 12.521620 8.551021 4.873367 2.089831 4.317756 2.295651 62.276524 4464.888008 7169.456 East Asia and the Paficic
174 Mali DHS 2018 Bambara 0.376063 0.356267 66.298032 3997.122754 53.737199 21.187866 ... 10.275852 7.833857 4.583490 5.904030 8.791907 0.793063 30.669522 6029.021745 19658.023 Sub-Saharan Africa
130 Guyana MICS 2019/2020 Other ethnicity/DK/Missing 0.006592 0.000000 0.000000 0.000000 44.117537 0.000000 ... 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.198953 1.557356 782.775 Latin America and the Caribbean
83 Gabon DHS 2012 Other 0.069695 0.111062 23.392190 43.069936 47.478026 32.666112 ... 7.110192 11.398305 8.269101 6.701042 7.847704 5.757293 8.474771 184.121009 2172.578 Sub-Saharan Africa
91 Gambia MICS 2018 Sarahule 0.203638 0.296093 59.977752 124.355645 49.367169 38.676186 ... 11.180899 5.214931 2.020000 3.244639 1.524079 0.206847 8.831479 207.336289 2347.696 Sub-Saharan Africa
41 Chad MICS 2019 Gorane 0.517011 0.577274 89.482903 776.671700 64.512217 16.070049 ... 8.136652 7.411264 3.956538 7.958233 7.943157 5.288613 5.442791 867.955408 15946.882 Sub-Saharan Africa
137 Kazakhstan MICS 2015 Other ethnic group 0.001611 0.001720 0.459870 14.305641 0.000000 70.668384 ... 5.795423 0.000000 0.000000 0.354503 5.440920 0.354503 16.768532 3110.802048 18551.428 Europe and Central Asia
18 Burkina Faso DHS 2010 Gourounsi 0.523424 0.478187 80.624264 724.623172 59.310508 18.106015 ... 9.351892 8.953515 4.134148 9.276997 7.840116 1.951105 4.422758 898.765622 20321.383 Sub-Saharan Africa
129 Guyana MICS 2019/2020 Mixed race 0.006592 0.007100 1.931164 2.777855 36.766958 36.747468 ... 2.310914 5.282043 4.563718 11.599562 13.860650 5.365067 18.376105 143.843552 782.775 Latin America and the Caribbean
38 Chad MICS 2019 Boulala/Medogo 0.517011 0.573660 88.006067 469.706504 65.184104 17.362271 ... 8.010858 7.892399 3.386179 8.150411 8.197027 4.296895 3.346865 533.720593 15946.882 Sub-Saharan Africa
34 Central African Republic MICS 2018/2019 Zandé/Nzakara 0.461348 0.440371 81.541115 145.487272 54.006010 17.638498 ... 10.271583 10.086051 9.258491 9.187658 9.648795 8.410939 3.760068 178.421980 4745.179 Sub-Saharan Africa
258 Sri Lanka SLDHS 2016 Other 0.011185 0.000000 0.000000 0.000000 0.000000 0.000000 ... 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.036497 7.782426 21323.734 South Asia

30 rows × 26 columns