Data Drives Improvement
Boston Beyond has a national reputation for being particularly data oriented. Every after school and summer program in their network participates in measurement, or data collection, to facilitate improvement in program practice and student skill development.
“All data collection focuses on answering three key questions – whom did we serve? Are we providing high quality experiences for students? Did students experience skill growth?" —Lisa Gomi Hui, Director of Measurement & Improvement
Boston Beyond’s four-person Measurement & Improvement team is responsible for collecting data from each of their 359 programs, cleaning the data, and presenting it so that programs, Boston Public Schools, the City of Boston, and researchers can get answers to these three key questions.
The Measurement & Improvement team’s challenge was to scale up their data collection, cleaning, and presenting for their rapidly growing network, going from 183 programs in 2017 to 359 programs in 2020 while maintaining their small team size. Hui explains:
“First, we needed get data back to our program partners more quickly. We wanted partners to be able to review their data and apply it to program improvements right away, but it was taking us up to four months to get the data report back to them. Our small team was busiest in the seasons when data was needed most (end of school year / beginning of summer, end of summer / beginning of school year).
“The Excel-based PDF data report we were using was outsourced to a contractor that was increasingly taking longer and longer to produce accurately. We hired a Data Architect dedicated to our data infrastructure and built a data dashboard using Microsoft Power BI, but we still had an enormous amount of data to clean and return within a few weeks. (Data included student enrollment, attendance, four different youth surveys, two different observation ratings, and a staff survey. As an example, one of the observation rating tools used at nearly all participating programs use has 394 data fields per observation – and we coordinate two observations during the school year and one during the summer.)
“Second, staff time was overwhelmingly being spent on coordinating measurement activities like survey administration and data collection, but not on improvement activities like reviewing data with partners.
“Given how onerous the data cleaning load was (especially with only Microsoft Excel and IBM SPSS at our disposal), we never had time to organize data review sessions with partners before the next school year or summer required our attention for measurement logistics.
“Lastly, we were struggling to meet the data reporting needs of all our different stakeholders. The Boston Public Schools, funders, and researchers all wanted data about the programs in our network but often in very specific, different arrangements. Some needed this data annually; others were more ad hoc. All of our program and student information is housed in Salesforce, which helped with producing basic reports. However, our stakeholders often wanted data presented in ways that required significant outside calculating or editing outside of Salesforce, which we were doing in Excel or SPSS. This work kept our staff away from the core work of measuring and improving programs.”
Four Ways Alteryx Has Helped Solve Boston Beyond’s Challenges
“In June 2019, our data architect told us about Alteryx. He had seen it used at his previous job by an analytics team and thought it could be a helpful tool to solve our problems. He was right,” said Hui.
When cleaning data with Microsoft Excel or IBM SPSS, we had to rely on point-and-click or SPSS syntax, and the limited data manipulation tools (e.g. VLOOKUP, Filters, Pivot Tables etc.) Cleaning data sources this way often took days of multiple staff people’s time. With Alteryx, once you built a workflow, the workflow itself took seconds.
Partly why data cleaning took so long was because mistakes in Microsoft Excel or IBM SPSS were often not undoable. When we made mistakes in data cleaning in the middle of the process that we couldn’t control-Z our way out of, we had to close the file and start over. We’d even resorted to saving the file every so often at intermediate stages, leading to cluttered and confusing folders. With Alteryx, we could edit the workflow and run the workflow as many times as we needed until we got our final desired output.
Although each member of the M&I team is comfortable with interpreting and sharing data, only our data architect knew programming languages to quickly edit data and maintain databases. Alteryx was like giving programming ability to those who did not know programming languages. The tool palette and the workspace made intuitive sense, and once we understood the tools, we could easily edit data as needed.
As our team grew and our program partner network grew, we needed to document processes better. With Excel or SPSS, we would have to write documentation in a separate file and the person reviewing would need to look at the file side-by-side with the documentation. With Alteryx, the entire process was laid out in the workflow – and with the Comment tool, you could clarify any parts of the process.
Measures are Only as Good as the Change They Inspire
“In the first two months of Alteryx, our data architect figured out how to solve our first challenge of cleaning data in an accurate and timely way for program partners. For every data source, he built an extensive workflow that led to a final, cleaned output – all field names consistent with past files, correct organization and program names, ID numbers, affiliated initiatives or research projects. Instead of taking days to clean the file (and have to do it over and over at the end of each semester), now we could take a given file and clean it in a matter of minutes and have it write directly into our SQL database for our data dashboard,” said Hui.