In a three-page memo to all 94 U.S. Attorneys' offices around the country, Holder said rising prison costs have resulted in reduced spending on law enforcement agents, prosecutors and prevention and intervention programs.
"These reductions in public safety spending require us to make our public safety expenditures smarter and more productive," the memo stated.
In some cases where a defendant is not an organizer, leader, manager or supervisor of others, "prosecutors should decline to pursue charges triggering a mandatory minimum sentence," Holder's memo stated.
In his speech to the ABA, the attorney general said "we need to ensure that incarceration is used to punish, deter and rehabilitate -- not merely to convict, warehouse and forget."
Holder said new approaches -- which he is calling the "Smart On Crime" initiative -- are the result of a Justice Department review he launched early this year.
The attorney general said that some issues are best handled at the state or local level and that he has directed federal prosecutors across the country to develop locally tailored guidelines for determining when federal charges should be filed and when they should not.
He said 17 states have directed money away from prison construction and toward programs and services such as treatment and supervision that are designed to reduce the problem of repeat offenders.
In Kentucky, legislation has reserved prison space for the most serious offenders and refocused resources on community supervision. The state, Holder said, is projected to reduce its prison population by more than 3,000 over the next 10 years, saving more than $400 million.
He also cited investments in drug treatment in Texas for non-violent offenders and changes to parole policies, which he said have brought about a reduction in the prison population of more than 5,000 last year. He said similar efforts helped Arkansas reduce its prison population by more than 1,400. He also pointed to Georgia, North Carolina, Ohio, Pennsylvania and Hawaii as states that have improved public safety while preserving limited resources.
San Francisco County District Attorney George Gascon applauded Holder's speech.
"It's obviously a big shift in policy," Gascon said. "Now let's see how the follow through works."
In a state experiencing severe prison overcrowding, Gascon has been advocating "alternative" sentencing of low-level drug offenders since taking office as district attorney in January 2011. He previously served as the city's police chief. Last week, the Supreme Court refused to delay the early release of nearly 10,000 California inmates by year's end to ease overcrowding at 33 adult prisons.
Praising Holder's efforts, Laura W. Murphy, director of the American Civil Liberties Union's Washington Legislative Office, said the attorney general "is taking crucial steps to tackle our bloated federal mass incarceration crisis."
Julie Stewart, president of Families Against Mandatory Minimums, said, "For the past 40 years, the Department of Justice, under both political parties, has promoted mandatory minimum sentencing like a one-way ratchet."
Former federal appeals court judge Timothy Lewis recalled that he once had to sentence a 19-year-old to 10 years in prison for conspiracy for being in a car where drugs were found. Lewis, a former prosecutor, said the teen, who was black, was on course to be the first person in his family to go to college. Instead, Lewis had to send him to prison as the teen turned and screamed for his mother.
"I am just glad that someone finally has the guts to stand up and do something about what is a pervasively racist policy," said Lewis, who is black.