Today the Taj, which draws millions of tourists a year, is surrounded by neat rectangles of manicured lawn.
"The gardens would have had to be trimmed back but those who saw them before spoke of how the greenery gradually revealed the mystery" of the Taj's stunning facade, Herbert told AFP in an interview.
Herbert's book was published recently to rave reviews in India with news magazine India Today calling it a "scholarly tour de force".
The viceroy was not alone in putting his stamp on the Indian landscape, and a horticultural legacy remains more than sixty years since the end of British rule.
Homesick colonials left their imprint on India's parks and gardens, many of which are still full of tidy, green lawns, trimmed hedges and flowerbeds with British blooms.
The British left "a lasting horticultural mark on India -- much as India did on them", writes Herbert, whose book was compiled from letters and diaries of British colonialists and official archives.
When the British first arrived in India in the 17th century, they discovered a continent brimming with strange but lush, exotic flowers that made them yearn for their carefully tended gardens back home.
"The farther from home they ventured, the more they longed for familiar cowslips and hollyhocks and Michaelmas daisies, for well-trimmed lawns and neat flowerbeds," says Herbert, professor emeritus of history at Mount Holyoke College in Massachusetts.
In India, where outbreaks of malaria, dysentery and other illnesses constantly brought death, many colonialists desired the comfort that a well-ordered garden could lend, she says.
For the British in India, "like tea, gardens offered colonials reassurance in situations of stress", she says.
The joy of recreating an English flower garden in a foreign landscape was recounted in rhapsodic terms by a military officer's wife, Edith Cuthell, in 1905.
"My violets are in bloom," Cuthell wrote in a letter home to her mother. "You cannot think how one treasures out here the quiet little 'home flower'. Dear little English flower," she said.
Wherever the British went, Herbert notes, they took with them, as part of their "cultural baggage", their love of gardens and their view about what a garden should look like -- "and nowhere was this more evident than in India".
Large houses drew inspiration from country estates back home with their "sweeping park, copses of trees and water" and bungalows had their "gravel paths, shrubs, flower beds and attempts at a lawn".
Usually, the garden was the responsibility of the woman, who was "terribly bored with her husband busy running India," says the book, which Herbert told AFP took her "the better part of 10 years" to write.
British women in India relied on the services of a gardener to do the actual groundskeeping.
With labour so cheap, sometimes the grass was planted blade by blade to create the lawn -- the essential part of any proper English garden -- and also cut painstakingly by hand.
But the tame landscapes created by the British were a far cry from the extraordinary beauty of Mughal gardens. Humayun's Tomb in New Delhi -- the inspiration of the Taj Mahal -- is a stunning example.
The Mughals, famed for their architectural splendours, are one of the Indian subcontinent's great dynasties and ruled from the early 16th to the mid-18th century.
The gardens of Humayun's Tomb with their pools, plants and trees are meant to symbolise paradise and are one of the earliest examples of a royal garden tomb in the Indian subcontinent, scholars say.
But they fell into neglect as Mughal power waned and were only restored a decade ago when the Aga Khan Trust for Culture and the Archaeological Survey of India brought back to life long-dormant fountains for the first time in some 400 years.
Water started flowing through narrow canals representing the four rivers of paradise described in the Koran and gardeners planted thousands of mango, lemon and pomegranate trees and sweet-smelling hibiscus and jasmine plants -- Mughal favourites.
Despite such splendour, the British stamp on gardens remains elsewhere in India.
The Taj has become a "bit of a template" for Indian official monuments and gardens, says Herbert, while English annuals and perennials have become the mainstay of many gardens in the subcontinent.
It is "one of the more benign legacies" of British rule in India, says Herbert.