The drug, enzastaurin, was being tested in lymphoma patients who were at high risk of relapse following chemotherapy treatment. Lymphoma is a cancer of the lymphatic system.
The decision to stop the drug's development will result in a second-quarter charge of about $30 million, but the company's 2013 forecast remained unchanged, Lilly said in a statement on Friday.
Lilly over the past decade has also tested the pill against a variety of other types of tumors, without success, including lung cancer and breast cancer. Its main remaining focus has been on lymphoma.
Cowen and Co had expected Lilly to seek marketing approval for the medicine this year as a lymphoma treatment, and for it to generate annual sales by 2016 of about $200 million, if approved for the condition. That would have made it a fairly modest product for Lilly.
Approval would have given a boost to Lilly's cancer program. The company's third-biggest medicine is Alimta, a treatment for lung cancer that pulls in $2.5 billion a year, but it could face generic competition in the United States by 2016.
Lilly on Friday said it continues with late-stage trials of two other experimental cancer drugs.
Lilly shares were little changed in morning trading.
(Reporting by Esha Dey in Bangalore and Ransdell Pierson in New York; Editing by Steve Orlofsky)