Researchers and experts said the report offered a valuable, if limited, snapshot of the program that was based on a one-year study of 1,700 students — 995 who were selected through a lottery to receive scholarship offers, and 776 who were not.
The program was established in 2004 during Mr. Boehner’s tenure as chairman of the House Committee on Education and the Workforce, and he proposed the legislation to reauthorize it in 2011 under the Scholarships for Opportunity and Results Act. Congress also mandated an independent evaluation of the program.
The report, released Thursday, was well underway before the 2016 election, but it comes as President Trump and Ms. DeVos press Congress to devote hundreds of millions of dollars to a voucher program as part of a $1.4 billion plan to expand school choice among private, parochial and public charter schools — a move that critics say would cripple the public education system.
Researchers said that the Washington program was not intended to make a case for or against a national voucher system. The city has a robust choice of charter and private schools, and public schools have significantly improved in the past decade — conditions not mirrored in other school districts.
“In D.C., it’s good to think of this as a study of value added to an environment that is already rich with school choice,” said Marsha Silverberg, who oversaw the evaluation for the Institute of Education Sciences.
Still, the results are sizable enough to conclude that students who were not selected for vouchers fared better academically.
Math scores among students who used the vouchers were roughly seven percentage points lower than students who were not selected. The negative academic effect was even more pronounced for students who were not attending a low-performing school when they were awarded the vouchers — their scores were 14.6 percentage points lower in reading and 18.3 percentage points lower in math — and for students in elementary school.
Switching schools had little effect on the achievement gap, the report found, but students in public schools did receive more instruction time each week in math and reading than students who used the private school vouchers. Private school proponents have often argued that comparing standardized test scores of public and private schools is unfair because public schools are held more accountable for test scores and tend to spend more time teaching tested subjects while private schools offer enrichment programs like art and music.
Still, Mark Dynarski, the president of Pemberton Research and the lead author of the study, said it was hard to speculate why the achievement gaps exist.
The report adds to mounting evidence that voucher programs across the country, which are often seen as an alternative to inferior public schools, are producing mixed academic results. Recent examinations of programs in Indiana, Louisiana, Ohio and Wisconsin have drawn similar conclusions.
On a positive note, families who participated in the Washington program reported that their private schools were safer by a large margin. They also had slightly higher satisfaction rates with their private schools, though researchers deemed that gap statistically insignificant.
Patrick J. Wolf, a University of Arkansas professor who served on the report’s advisory committee, said the findings were more telling than the narrower lens of test scores.
Mr. Wolf, who led the research team that conducted the scholarship program’s first evaluation in 2010, said many studies showed that families were trading academic rigor for assurances that their children would be safe at school.
Credit Maddie McGarvey for The New York Times
“If you give a parent a choice between a child who is safe or doing better in math, they’re going to choose safe every single time,” Mr. Wolf said. “It’s the hierarchy of human need.”
The new analysis fueled criticism from congressional Democrats who vehemently oppose public funding for private schools and have derided Ms. DeVos’s affinity for the idea.
Ms. DeVos, a billionaire, has advocated school choice expansion for decades in her home state, Michigan. As education secretary, she has praised a longstanding tax credit that funds vouchers in Florida and a recent expansion of vouchers in Arizona.
Senator Patty Murray of Washington, the ranking Democrat on the Senate Health, Education, Labor and Pensions Committee, said that “when Secretary DeVos’s own department’s independent research office tells her that siphoning taxpayer dollars into private schools has a negative impact on students, it’s time for her to finally abandon her reckless plans to privatize public schools across the country.”
Ms. DeVos pledged to continue fully funding the Washington voucher program.
Serving Our Children, the organization that manages the program, said it had provided individual scholarship awards of up to $12,679 for high school and $8,452 for elementary and middle school this school year. About 1,150 students use the program, attending 41 schools; 81 percent are black, and more than half receive some kind of public assistance.
Ms. DeVos pointed out that the district’s public school system had not suffered, but had improved, since the program began.
But she did say that private schools needed to be held accountable for better outcomes for students, the same demand that she has made on public schools. “These schools need to improve upon how they serve some of D.C.’s most vulnerable students,” Ms. DeVos said in a statement.
Representative Virginia Foxx of North Carolina, the chairwoman of the House Committee on Education and the Workforce, praised the scholarship program for its transparency and accountability but called the evaluation’s results “disappointing.”
“Those who operate and participate in this important program must step up and deliver the change that is necessary to ensure students excel from Day 1,” she said.