Home > Uncategorized > Data-driven decision making takes a pratfall

Data-driven decision making takes a pratfall

April 16, 2012

Delaware school districts have launched a groundbreaking and promising plan to promote parent engagement by tracking usage data for one of Delaware’s premier data systems – Sungard’s Home Access Center (part of eSchoolPLUS).

But despite Delaware’s commitment to data-driven decision making, the usage reports available from eSchoolPLUS are not detailed enough to support credible decision making for this plan.

Delaware Department of Education data gurus should be urgently requesting that Sungard upgrade its product to provide these reports, or to provide an interim workaround. In the meantime DDOE should be taking what steps it can to create custom reports that are more detailed. It’s not too late to improve the reports for next year, as long as action is taken this summer.

District plans
In their district success plans, 13 out of 19 districts have committed to increasing the numbers of parents who use the Home Access Center. HAC is the web portal for parents to look up assignments and grades, among other things. It is part of Delaware’s student information system (eSchoolPLUS) provided by Sungard.

The districts are on the right track here, because HAC is a powerful enabler and indicator of parent engagement. The more parents use it, the better. Here’s one quote from the Appoquinimink success plan:

Increased parent use of Home Access system creates awareness and has the potential impact to increase parent expectations for student achievement.

I’ve been advocating for a HAC usage report for some time now. Districts, schools, and parent advocates all stand ready to reach out to families and students to put this valuable tool (HAC) into more hands. With a reliable usage report, we can measure and fine-tune the results of their outreach, and judge the effectiveness of different strategies. The district success plans lay out specific tactics they plan to take to increase parental engagement through HAC. But Sungard does not provide a report for HAC usage.

There is a way for DDOE administrators to create a custom report, but the underlying data infrastructure is so rudimentary the basic report carries very little information and is virtually useless for the district’s purpose.. In my opinion, the inability for a modern web product to properly report its own usage is a defect.

Everybody acknowledges that parent engagement is one of the most important factors for student success. We need to be able to rely on our information systems to support our efforts to promote parent engagement.

The eSchoolPLUS system and HAC are proprietary, and Sungard is unwilling to share details about how the usage reporting mechanism works. I asked about this very thing on Sungard’s public blog. In response, I got an email that informed me Sungard’s policy was not to respond to parents about their products, and my message was forwarded to DDOE. And now, let’s just say discussions with DDOE on usage reports are proceeding slowly.

But I have gathered technical information from a variety of public sources, and have reached some conclusions and some recommendations. What follows is my informed guesswork on the technical details, so if you have any better (verifiable) information please add a comment, or email me at mike01@seventhtype.com.

[...]

How eSchoolPLUS stores usage data
When you log in, eSchoolPLUS stores the date and time of your login in your account profile. The next time you log in, the date and time of your login is stored again – overwriting the previous timestamp. So all the timestamp indicates is the last time you logged in. It provides no information about your ongoing level of use.

It is relatively easy for DDOE to count these timestamps and create a report counting all the users who have ever logged in. The database is cleared out at the beginning of the school year, so really this count is simply a “Year To Date” count of all the users who have logged in this year, even if only once. And in fact, this is what DDOE gives to districts who ask for HAC usage reports.

Here are the Christina School District’s usage commitments, from their District Success Plan (click to enlarge):

Data quality problem #1
But a YTD usage report is simply not detailed enough for data-driven decision making. According to Sungard, the parent who logs in once in September and then shuts it down in bewilderment never to return, counts just the same as the parent who logs in daily or weekly and gets real benefit from the system. To measure meaningful usage levels, we need data points taken more frequently throughout the year, preferably daily.

To create an enhanced report, DDOE could improve on the YTD report by counting the timestamps daily and storing the counts somewhere else. Then they could run a report showing the unique user login count for each day. The daily counts could be reported as a bar and/or line chart showing daily, monthly, and YTD usage, with the counts also available as a percentage of active accounts. Then we’d be getting somewhere closer to a proper enterprise usage report. But DDOE has so far shown no indication it is willing to create this report.

To be fair, creating this type of report would require assigning a technical resource to complete it. But it is within the capabilities of a junior programmer, and probably wouldn’t take that long to at least output it to a spreadsheet. Creating a web reporting page would take a little longer, but not unreasonably so.

Data quality problem #2
Due to its current technical design, eSchoolPLUS cannot accurately associate a login with the school it came from. This is insufficient data for schools or parent groups who want to measure the results of their outreach efforts to increase HAC usage. For parents with one child in one school, reporting is accurate – because the login can be associated with the child’s school. But for parents with students in two different schools in the same district, first you log in, then you see a screen that asks you which child’s record you want to see.

Presumably your login timestamp is already recorded before you choose the child, so there is no way for eSchoolPLUS to associate a school with the login. So the more parents with children in different schools, the more skewed the report would be.

(I have slightly less confidence in this description, so I would be happy to hear that I am wrong and that Sungard supports tracking usage by school as well).

Conclusion and recommendations
The underlying data structure and lack of reporting infrastructure for HAC shows the age of Sungard’s eSchoolPLUS, which began its life over a decade ago as a (non-web) product called Pentamation, with the web portal (HAC) retrofitted later.

But now that HAC is here and is a valuable part of our school system, it is time for Sungard to provide a modern reporting capability sufficient to support data-driven decision making. And Sungard should do this without requiring any additional fees.

If not, then it’s time to start planning for our own Parent Dashboard with full reporting capability, among other enhancements.

The Delaware Department of Education should contact Sungard immediately and explain this situation, insist on enhanced reporting, and report the results of this conversation to the public.

In the meantime, DDOE should develop its own daily report capturing the existing data each day, and publish monthly reports including bar and/or line charts showing daily, monthly, and annual usage, broken down by district. DDOE should also look at available options for associating logins with a school and reporting usage by school as well.

A modern usage report should show:

  • Usage (number of unique logins) per day, per week, per month, and per marking period.
  • Usage by number, by percent of all registered accounts, and by percent of all eligible users.
  • Usage by school, by district, and for the state.
  • Graphical reports including bar graphs, line graphs, and other graphs.
  • Account separately for parent and student logins

These reports should show a comparison of all districts and be released to the public.

The District success plans acknowledge that they don’t know what their current HAC usage rates are, but are committed to recording the baseline usage in this school year, and increasing it to their target by each year until 2014. Baselines are due to be recorded this year. Unfortunately, the only baseline available to us right now is the YTD login count with no school association. Let’s develop those reports and do better next year.

About these ads
Comments are closed.
Follow

Get every new post delivered to your Inbox.

%d bloggers like this: