An Amazon spokeswoman said the company is glad that workers chose to keep a “direct relationship with Amazon,” which the company sees as best for employees and customers.

“We will continue to work directly with our teammates in Albany, as we do everywhere,” the spokeswoman said.

“We are filled with resolve to continue and expand our campaign for fair treatment for all Amazon workers,” Chris Smalls, president of the Amazon Labor Union, said Tuesday. “It isn’t a loss; it’s an ongoing battle,” he said.

Amazon shares rose more than 2% Tuesday while the tech-heavy Nasdaq Composite Index rose less than 1%.

The election at the facility, named ALB1 and located at Castleton-on-Hudson, N.Y., was the fourth this year at an Amazon warehouse. The ALB1 center is much smaller than the facility that unionized in Staten Island, named JFK8. About 950 people were eligible to vote at ALB1, compared with the roughly 8,300 eligible voters at the time of the JFK8 election in April. More than 600 workers cast ballots in the ALB1 election.

Workers at ALB1 make a base pay of around $17 an hour after a recent wage increase, according to the company. Amazon recently raised starting wages for front-line employees by $1 to an average of $19 an hour across the country. Amazon has emphasized its pay and health benefits relative to other similar jobs as it has faced unionization attempts, and company leaders have said they see unions as a potential impediment to flexibility.

Union organizers have sought to negotiate with Amazon over pay and working conditions, including safety measures and job security. Federal labor data has shown Amazon warehouses having more injuries than the national average. The company has tried to address injuries through education courses and other tactics and said it reports injuries better than rival companies. Amazon warehouses have long had high turnover due to a variety of factors, including burnout and Amazon’s up-and-down staffing needs at facilities, according to workers.

The union in May lost a second election at a smaller company facility in Staten Island and has been at odds with Amazon over its victory at JFK8 after the company contested the results of that vote. A federal labor official recently recommended that the vote be upheld, but the results haven’t yet been certified.

Heather Goodall, an employee and organizer at the ALB1 facility near Albany, said Amazon suggested to workers that their pay and benefits could suffer if the union represents them. Amazon said the company holds meetings to help staff “understand the facts” about joining a union and the election process itself. Amazon in recent months has also fired some union organizers. The company has said that employees were let go after violating internal policies.

Those who voted against unionizing said they prefer to maintain the status quo. Yari Reyes, who has worked at ALB1 for about two years, said she wasn’t convinced the union could deliver better pay and benefits.

“Having a union come in and disturb the communication we have with leadership, why have that?” she said. “Bringing a union in would have just divided us more.”

Amazon and many of its workers remain at odds. Groups of workers at multiple facilities said they went on strike last week, including at a company air hub in Southern California, to protest pay and other issues. Some workers have said that Amazon’s recent pay increases didn’t make up for the uptick in consumer prices many Americans have seen this year.

The company earlier this month suspended more than two dozen workers at the JFK8 facility in Staten Island after night-shift employees refused to work after a cardboard compactor at the center caught fire earlier in the day. Staff members said they raised concerns about working despite the earlier fire, but that managers insisted they would have to work. Amazon said the New York City Fire Department certified the building safe before they asked night-shift workers to complete their shift.

A small fire also broke out the same week at ALB1. Amazon said the site was evacuated and employees were sent home with pay.

Write to Sebastian Herrera at Sebastian.Herrera@wsj.com