Feature Engineering and Exploratory Data Analysis

 



For this week, I did Feature Engineering and Exploratory Data Analysis (EDA), using the Alabama data as a prototype. I created the the custom features that define the originality of the project, and then did statistical and visual analyses to validate my core hypotheses.

In the Feature Engineering phase, I utilized a Python program to merge the block-group-level broadband speed data (summarized to show max speed) with the median income data and the U.S. Census shapefiles, successfully creating a unified GeoDataFrame. From this GeoDataFrame, I derived three critical features:

  1. The Socioeconomic Vulnerability Index (SVI): Calculated by inverting the normalized median income, this score quantifies digital inequity risk, with values closer to 1 indicating high vulnerability (low income).

  2. Neighbor Average Speed: This spatial feature captures the average broadband speed of all adjacent block groups, serving as a powerful proxy for regional infrastructure investment.

  3. Neighbor Average SVI: This feature indicates whether a block group is situated within a larger cluster of vulnerable areas, showing that the digital divide is a widespread community issue.

The Exploratory Data Analysis (EDA) phase yielded powerful, data-driven insights, primarily from the correlation matrix. This analysis confirmed my central hypothesis: that the digital divide is driven by socioeconomic factors, showing a modest correlation between a block group’s SVI and its Max Down Speed (−0.11). Critically, the EDA revealed that my engineered features are the strongest predictors: the Neighbor Average Speed showed a strong positive correlation (+0.62) with a block group's own speed, validating the project's focus on spatial context. 

Comments

Post a Comment

Popular posts from this blog

Wrapping Up: A Geospatial Dive into the U.S. Broadband Divide

Image
         For my MSDS692 Data Science Practicum project, I decided to tackle a topic close to my professional background: the U.S. Broadband Divide. My goal was to move beyond simple maps and explore whether socioeconomic factors like income and race contribute to disparities in internet access. I wanted to visualize these patterns and ultimately see if we could predict underserved areas using data science.      Leveraging large datasets from the FCC's Broadband Data Collection (providing provider speeds) and the US Census/ACS (offering income, population demographics, and geographic boundaries), I analyzed broadband availability at a granular Census Block Group level across the nation.      The core data science workflow involved several key steps. First came the significant task of downloading, cleaning, and aggregating hundreds of files to create unified national datasets for speed, income, and minority population percentage at ...
Read more

MSDS Practicum Project - US Broadband divide

Image
  The Term US Broadband divide indicates the unavailability of broadband internet to all americans. According to FCC, 100 mbps speed is the threshold to be called as broadband. The site Ookla.net collects speed tests and shows 60% of the tests > 100 mbps download speeds The data from the FCC- BDC as of December 2024 demonstrates a steady increase in locations with access to high-speed internet. According to the FCC, an additional one million locations gained access to 100/20 Mbps speeds between June and December 2024. Additionally, nearly 7 million additional locations gained access to even faster gigabit speeds (1 Gbps download and 100 Mbps upload) in the same time frame. While the overall trend is positive, the persistent percentage of unserved and underserved locations represents a significant population that is still being left behind The good news is over the years with expanding cable infrastructure and advancement in other technologies - Terrestrial fixed wireless, Cell...
Read more

Downloading data from census.gov's ACS

Image
       The census.gov website has ACS ( American Community survey ) data that has a wealth of information about demographics, median household income etc. It provides API support to download the data. In this tutorial, I will explain how to download the data using API calls, and also creating a python script to automate downloading data with multiple API calls. step1 : get a API key. The API key is needed to make the API calls. sign up and get the key in this URL:  https://api.census.gov/data/key_signup.html step2 : assemble the API call URL. depending upon the information needed, we have to assemble the URL needed to do the API call.  for example: https://api.census.gov/data/2023/acs/acs5?get=NAME,B19013_001E&for=block%20group:*&in=state:36%20county:119&key=xxxxxxxxxxxxxxxxx in the above API Call url,  B19013_001E is the data point for household median income.  in=state:36%20county:119 represents NY state and Westchester county. &...
Read more